Working with Directories (os, pathlib) in Python

Python gives you two genuinely different ways to work with the file system: the older, string-based os/os.path modules, and the modern, object-oriented pathlib, introduced in Python 3.4. This article covers python pathlib vs os side by side — creating, listing, navigating, and inspecting directories — with honest guidance on where os is still the tool you actually need.

Introduction: Two Ways to Talk to the File System

The core philosophical difference

os/os.path treats file paths as plain strings, and every operation — joining, splitting, checking existence — is a separate function call taking that string as an argument. pathlib wraps a path in a Path object, giving you methods and operators directly on the path itself, rather than functions you call on a string.

import os
import os.path

# os.path style — functions operating on strings
full_path = os.path.join("data", "reports", "summary.csv")

from pathlib import Path

# pathlib style — a Path object with its own behavior
full_path = Path("data") / "reports" / "summary.csv"

Both produce the same practical result, but the second reads more naturally, and — as this article covers — genuinely scales better once you start chaining several operations together.

What this article covers

Creating and checking directories, listing and navigating their contents, building and inspecting paths, and honest guidance on when os remains the correct tool rather than pathlib — this isn't a "pathlib is strictly better" article; both have a real place.

Creating and Checking Directories

The os way
import os

# A single directory
os.mkdir("data")

# Nested directories — os.mkdir() alone would fail if parent folders don't exist
os.makedirs("data/reports/2026")

# Checking existence first, to avoid an error on a re-run
if not os.path.exists("data"):
    os.mkdir("data")

os.mkdir() creates exactly one directory, and raises an error if any parent directory in the path doesn't already exist. os.makedirs() creates the full chain of nested directories — but both will raise FileExistsError if the target directory already exists, which is why the manual os.path.exists() check is often needed before calling either.

The pathlib way
from pathlib import Path

Path("data/reports/2026").mkdir(parents=True, exist_ok=True)

This single line handles both nested creation (parents=True, equivalent to os.makedirs()) and silently succeeding if the directory already exists (exist_ok=True) — no manual existence check needed at all, and no risk of an error interrupting a script that's simply being run a second time against a directory structure that's already partially set up.

Side by side
# os — three lines, a manual existence check, and separate functions
import os
if not os.path.exists("data/reports/2026"):
    os.makedirs("data/reports/2026")

# pathlib — one line, handles both concerns directly
from pathlib import Path
Path("data/reports/2026").mkdir(parents=True, exist_ok=True)

This is a genuinely representative example of the ergonomic gap between the two approaches — pathlib collapses what would otherwise be several lines of os boilerplate into a single, self-documenting call.

Listing and Navigating Directory Contents

The os way
import os

# A flat list of names in one directory
names = os.listdir("data")
print(names)   # ['reports', 'config.json', 'notes.txt']

# Recursively walking an entire directory tree
for root, dirs, files in os.walk("data"):
    for file in files:
        print(os.path.join(root, file))

os.listdir() gives you a flat list of names — files and subdirectories mixed together, as plain strings, with no distinction between the two without a further check. os.walk() recursively traverses an entire directory tree, yielding a (root, dirs, files) tuple for every directory it visits — genuinely useful, but noticeably more verbose to work with directly.

The pathlib way
from pathlib import Path

# One level, lazily
for entry in Path("data").iterdir():
    print(entry)   # data/reports, data/config.json, data/notes.txt

# Pattern-based, one level
for csv_file in Path("data").glob("*.csv"):
    print(csv_file)

# Pattern-based, recursive — searches every subdirectory too
for csv_file in Path("data").rglob("*.csv"):
    print(csv_file)

Path.iterdir() is a lazy iterator (mirroring the generator-based lazy evaluation covered in the earlier generators article) over one directory level, yielding full Path objects rather than plain name strings. Path.glob() and Path.rglob() (recursive glob) let you search by pattern directly — rglob("*.csv") finds every CSV file anywhere below a given folder, at any depth, in a single, readable call, without you needing to manually walk the tree and filter yourself.

A practical example: finding files by pattern and type
from pathlib import Path

reports_dir = Path("data/reports")

csv_files = [
    entry for entry in reports_dir.rglob("*.csv")
    if entry.is_file()
]

for file in csv_files:
    print(file)

.is_file() and .is_dir() are methods directly on the Path object itself, letting you filter cleanly — genuinely useful since glob()/rglob() patterns can technically match directories too, if a directory name happens to match the pattern.

Building and Inspecting Paths

The os way
import os

# Joining path components, staying cross-platform
full_path = os.path.join("data", "reports", "summary.csv")

# Pulling out pieces
filename = os.path.basename(full_path)         # summary.csv
name, extension = os.path.splitext(filename)   # summary, .csv

os.path.join() correctly handles the difference between Windows path separators (\) and Unix ones (/), which is exactly why you should always use it (or the pathlib equivalent) rather than manually concatenating path strings with + or hardcoding a separator character yourself.

The pathlib way
from pathlib import Path

full_path = Path("data") / "reports" / "summary.csv"

print(full_path.name)      # summary.csv
print(full_path.stem)      # summary
print(full_path.suffix)    # .csv
print(full_path.parent)    # data/reports
print(full_path.parts)     # ('data', 'reports', 'summary.csv')

The / operator joins path components naturally, reading almost exactly like the resulting path itself. .name, .stem, .suffix, .parent, and .parts give you every commonly needed piece of a path as plain attributes, rather than requiring separate function calls per piece — genuinely more concise once you need more than one of these values from the same path.

Chained manipulations: swapping an extension
from pathlib import Path

path = Path("report.csv")
new_path = path.with_suffix(".json")
print(new_path)   # report.json

The equivalent with os.path requires manually splitting the extension off and reassembling the string yourself:

import os

path = "report.csv"
name, _ = os.path.splitext(path)
new_path = name + ".json"
print(new_path)   # report.json

Both reach the same result, but .with_suffix() expresses the intent directly — "give me this same path, with a different extension" — in a single, self-documenting call, rather than a manual split-and-reassemble sequence.

A cross-platform note

Both os.path and pathlib correctly handle Windows-versus-Unix path separators under the hood — this isn't a difference between them. The genuine difference is ergonomics: pathlib's object model makes chained manipulations — join, then inspect, then modify — noticeably more concise and more readable, since each step is a method or attribute directly on the same object, rather than a fresh function call wrapping a plain string each time.

When os Is Still Necessary, and Best Practices

What pathlib deliberately doesn't cover

pathlib is specifically a path-and-filesystem-traversal library — it deliberately doesn't attempt to cover several other things os handles that are genuinely unrelated to path manipulation itself:

  • Environment variables — os.environ, for reading and setting environment variables, has no pathlib equivalent at all, since it's not about paths.

  • Process management — os.getpid(), os.system(), and related process-control functions remain squarely os (or subprocess) territory.

  • Low-level permissions and file descriptors — some lower-level operations still require os directly, even though pathlib's Path objects cover the majority of everyday file and directory work.

Practical guidance

Use pathlib for nearly all path manipulation and file/directory traversal — creating directories, listing contents, joining and inspecting paths, pattern-based file searching. It's the more modern, more readable choice for essentially everything covered in this article. Reach for os (or subprocess) specifically when you need OS-level environment variables, process control, or the handful of lower-level operations pathlib doesn't attempt to cover at all. This isn't "pathlib is strictly better and os is obsolete" — it's "these two tools cover genuinely different, only partially overlapping territory," and knowing which one to reach for depends entirely on what you're actually trying to do.

A real-world pattern: a ProjectPaths helper class

For any project of meaningful size, centralizing your directory structure in one place — rather than scattering Path("data") / "reports" throughout your codebase — pays off considerably:

from pathlib import Path

class ProjectPaths:
    root = Path(__file__).parent
    data = root / "data"
    reports = data / "reports"
    c root / "config.json"

    @classmethod
    def ensure_directories(cls):
        cls.data.mkdir(exist_ok=True)
        cls.reports.mkdir(parents=True, exist_ok=True)

ProjectPaths.ensure_directories()
print(ProjectPaths.reports)   # /your/project/data/reports

This is a genuinely useful pattern combining several concepts from earlier articles — class attributes (covered in the earlier instance vs. class variables article) holding shared, project-wide path constants, and a @classmethod (covered in the earlier static and class methods article) handling setup logic — giving your whole project one clear, centralized, single source of truth for where everything lives, rather than that logic being duplicated or subtly inconsistent across multiple files.

Bridging to libraries expecting plain strings

Some older libraries and APIs still expect a plain string, not a Path object. Two ways to bridge this:

from pathlib import Path

path = Path("data") / "reports" / "summary.csv"

print(str(path))          # data/reports/summary.csv (or data\reports\summary.csv on Windows)
print(path.as_posix())     # data/reports/summary.csv — always forward slashes, even on Windows

str(path) gives you the platform-native string representation. .as_posix() always gives you forward slashes, regardless of the actual operating system — genuinely useful when you need a consistent, predictable string representation (for logging, or for a config file meant to be portable across platforms), independent of whichever OS the code happens to be running on at the time.

PREVIOUSNEXT LESSON