Working with the with Statement in Python

You've used the python with statement constantly throughout this series — with open(...) as f: — without necessarily knowing what makes it work. python context managers are the mechanism behind it, and this article covers the full picture: the __enter__/__exit__ protocol, the simpler generator-based shortcut, real-world uses beyond files, and how to manage multiple resources cleanly.

Introduction: The Problem with Solves

The fragile pattern it replaces

Before with, safely pairing setup and cleanup code meant something like this:

f = open("data.txt")
try:
    c f.read()
finally:
    f.close()

This works, but it's fragile in a specific way: it relies on you, the developer, remembering to write the try/finally correctly, every single time, for every resource that needs cleanup. Miss the finally, or put the wrong code inside it, and a resource can leak — a file left open, a lock never released — especially if an exception occurs somewhere unexpected.

The core guarantee

with formalizes this pattern into something Python enforces automatically. When you write with some_object as x:, Python calls some_object.__enter__() to set up the new runtime context, and — critically — calls some_object.__exit__() when the block ends, no matter how it ends: normally, via an early return, or because an exception was raised partway through.

with open("data.txt") as f:
    c f.read()
# f.close() is guaranteed to have been called here, regardless of what happened above
A familiar example, generalized

You've typed with open(...) as f: throughout this series without necessarily thinking about the mechanism behind it. This article's core point: that same pattern generalizes far beyond files — to database connections, locks, temporary state changes, timers, and much more, all built on exactly the same underlying protocol.

The Protocol Behind with: enter and exit

How the as variable gets its value

If you provide a target variable with as, its value comes from whatever __enter__() returns:

with open("data.txt") as f:
    # f is exactly what open(...).__enter__() returned

__exit__(), meanwhile, receives details about any exception that occurred inside the block — its type, its value, and its traceback — as three separate arguments, letting __exit__() inspect (and potentially respond to) whatever went wrong.

Writing a first class-based context manager
class Timer:
    def __enter__(self):
        import time
        self.start = time.time()
        return self   # this becomes the 'as' value

    def __exit__(self, exc_type, exc_value, traceback):
        import time
        elapsed = time.time() - self.start
        print(f"Elapsed: {elapsed:.4f} seconds")
        return False   # don't suppress any exception

with Timer() as t:
    total = sum(range(1_000_000))

print(total)
# Elapsed: 0.0234 seconds
# 499999500000

__enter__() runs first, recording the start time and returning self (which becomes t). The code inside the with block runs next. __exit__() runs automatically once the block finishes — printing the elapsed time — regardless of whether the block completed normally or raised an exception along the way.

The suppression mechanism

__exit__()'s return value has special significance: if it returns something truthy, any exception that occurred inside the block is suppressed — treated as handled, and not propagated any further. Returning False (or None, which is the default if you don't return anything) lets the exception propagate normally, exactly as if the with block weren't there at all.

class SuppressValueError:
    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type is ValueError:
            print(f"Suppressing: {exc_value}")
            return True   # suppress this specific exception
        return False       # let anything else propagate normally

with SuppressValueError():
    raise ValueError("This gets caught")
    print("This line never runs")

print("Program continues normally")
# Suppressing: This gets caught
# Program continues normally

Suppressing exceptions should be rare and deliberate. Silently swallowing errors inside __exit__() without a genuinely good, specific reason creates exactly the same category of problem covered in the earlier exception-handling articles — real bugs disappearing silently, rather than surfacing where they can actually be diagnosed and fixed.

The Easier Path: contextlib.contextmanager

Why a full class often feels like overkill

Writing an entire class with two separate dunder methods for simple, straightforward setup/teardown logic is genuinely more boilerplate than the task usually warrants. Python's contextlib.contextmanager decorator offers a considerably more concise alternative.

The generator-based shortcut
from contextlib import contextmanager
import time

@contextmanager
def timer():
    start = time.time()
    yield              # everything before this is __enter__; everything after is __exit__
    elapsed = time.time() - start
    print(f"Elapsed: {elapsed:.4f} seconds")

with timer():
    total = sum(range(1_000_000))
# Elapsed: 0.0231 seconds

Everything before yield runs as the setup logic (equivalent to __enter__); everything after yield runs as the cleanup logic (equivalent to __exit__) — all without writing a single dunder method, or a class at all. This is precisely the same generator mechanics covered in the earlier generators article, applied specifically to context management.

A practical example with try/finally around yield

To guarantee cleanup runs even if the code inside the with block raises an exception, wrap the yield in try/finally:

from contextlib import contextmanager
import tempfile
import shutil

@contextmanager
def temporary_directory():
    path = tempfile.mkdtemp()
    try:
        yield path   # this becomes the 'as' value
    finally:
        shutil.rmtree(path)   # guaranteed cleanup, even if the block raises

with temporary_directory() as tmp_dir:
    print(f"Working in {tmp_dir}")
    # do work inside tmp_dir...
# tmp_dir is automatically removed here, whether or not an error occurred above

Whatever value is passed to yield becomes the as value inside the with block — here, path becomes tmp_dir. The try/finally around yield ensures shutil.rmtree(path) runs regardless of what happens inside the with block, mirroring exactly the guarantee __exit__() provides in the class-based version.

Real-World Uses Beyond Files

Database transactions
from contextlib import contextmanager

@contextmanager
def transaction(connection):
    try:
        yield connection
        connection.commit()      # only runs if the block completed without error
    except Exception:
        connection.rollback()    # runs if anything inside the block raised
        raise

with transaction(db_connection):
    db_connection.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1")
    db_connection.execute("UPDATE accounts SET balance = balance + 100 WHERE id = 2")

This is a genuinely common, practical pattern: if both database operations succeed, the transaction commits; if either one raises an exception, the transaction rolls back instead, and the original exception is re-raised (via the bare raise) so the caller still sees that something went wrong.

Temporarily changing state and reliably restoring it

Context managers are an excellent fit for any "set something, do work, then restore it" pattern — working directories, environment variables, logging verbosity:

import os
from contextlib import contextmanager

@contextmanager
def change_directory(path):
    original = os.getcwd()
    os.chdir(path)
    try:
        yield
    finally:
        os.chdir(original)   # guaranteed restoration, even if an error occurs

with change_directory("/tmp"):
    print(os.getcwd())   # /tmp
print(os.getcwd())   # back to the original directory, automatically

Without this pattern, temporarily changing global or shared state and reliably restoring it afterward — especially in the presence of a possible exception — requires exactly the same careful try/finally discipline that with exists to make automatic and hard to forget.

Standard-library helpers worth knowing

contextlib.suppress() — covered in the earlier multiple exceptions article — offers a clean, explicit way to ignore specific, expected exceptions:

from contextlib import suppress

with suppress(FileNotFoundError):
    os.remove("maybe_exists.txt")

contextlib.closing() — for objects that have a .close() method but don't natively support with themselves:

from contextlib import closing
from urllib.request import urlopen

with closing(urlopen("https://example.com")) as page:
    c page.read()
# page.close() is called automatically, even though urlopen()'s result
# doesn't implement __enter__/__exit__ on its own

closing() wraps any object with a .close() method, giving it with-compatible behavior — genuinely useful for older or third-party objects that predate, or simply never implemented, the full context manager protocol themselves.

Managing Multiple Resources: Modern Syntax and ExitStack

Parenthesized syntax for several context managers at once

Since Python 3.10, you can cleanly open several context managers in a single with statement using parentheses, spanning multiple lines readably:

with (
    open("input.txt") as infile,
    open("output.txt", "w") as outfile,
):
    for line in infile:
        outfile.write(line.upper())

Both files are correctly closed when the block exits, regardless of how it ends — this parenthesized form is functionally equivalent to nesting two separate with statements, but reads more cleanly, especially once you're managing more than two resources at once. (Before 3.10, the equivalent required either nested with statements or a single line joined with commas, without the parentheses — the parenthesized form is purely a readability improvement introduced later.)

ExitStack for a dynamic number of resources

Sometimes you don't know in advance how many context managers you'll need to open — say, a variable-length list of files, or resources entered conditionally based on runtime logic. contextlib.ExitStack handles exactly this case:

from contextlib import ExitStack

filenames = ["a.txt", "b.txt", "c.txt"]

with ExitStack() as stack:
    files = [stack.enter_context(open(name)) for name in filenames]
    for file in files:
        print(file.read())
# every file is correctly closed here, regardless of how many there were

stack.enter_context(...) registers each context manager with the stack, and when the ExitStack itself exits, every registered context is cleaned up automatically — in LIFO order (last registered, first cleaned up), mirroring how nested with statements would unwind if you'd written them explicitly. This is genuinely valuable when the exact number, or even the presence, of certain resources isn't known until runtime — something a fixed, hardcoded with (a, b, c): statement simply can't express.

Quick closing guidance

  • Reach for the @contextmanager decorator first for simple, straightforward setup/teardown cases — it's more concise, and covers the overwhelming majority of everyday context manager needs without any class boilerplate.

  • Use a full class-based context manager when you need to maintain meaningful state across multiple methods, or need fine-grained control over exception handling and suppression logic that goes beyond what a simple try/finally around a yield comfortably expresses.

  • A brief note on async contexts: if you're working with asyncio (covered in a later article in this series), async with is the asynchronous equivalent of everything covered in this article, built on __aenter__/__aexit__ instead of __enter__/__exit__ — the same underlying philosophy, adapted for coroutine-based resources like async database connections or network sessions.

PREVIOUSNEXT LESSON