Python Context Managers and the `with` Statement: Deterministic Resource Management

A leaked connection, a lock that never released, a transaction that committed on the wrong path. None of these are exotic failures. They are ordinary failures that happen when cleanup is treated as a line of code rather than a contract.

I have written try/finally blocks that looked correct and were not. The cleanup line sat after the return. The lock was acquired in one function and released in another that never ran. The connection closed on the happy path and leaked on the exception path that a retry loop quietly swallowed. Every one of those bugs was a context manager waiting to be written.

Here is the tension that makes this topic worth an article. The with statement is sold as "automatic cleanup." It provides no automatic anything. It provides two callbacks and one precise rule about when they fire and what their return value means. Everything else — what gets acquired, what gets released, whether an exception is allowed to continue — is a contract you write.

The invariant worth memorizing: after __enter__ returns successfully, __exit__ runs exactly once when the block exits, and its return value is the only thing that decides whether an exception continues.

What the with Statement Actually Compiles To

The context manager protocol is two methods.

__enter__(self) runs when execution enters the block. Its return value is what as binds. If you write with open(path) as f:, then f is whatever open(path).__enter__() returned.

__exit__(self, exc_type, exc_value, traceback) runs when execution leaves the block — on a normal fall-through, on return, on break, on continue, and on a raised exception. The three arguments describe the exception that caused the exit. On a clean exit, all three are None.

That last detail matters more than it looks. Code that inspects exc_type without handling the None case will break on the success path, which is the path you test least.

The with statement is a structured form of try/finally where the finally clause receives the exception triple instead of running blind. The call order for a block that raises looks like this:

1. __enter__ runs and returns a value.
2. The body executes.
3. The body raises.
4. __exit__(exc_type, exc_value, traceback) is called with the live exception.
5. If __exit__ returns a truthy value, the exception is suppressed. If it returns None or False, the exception propagates.

Step 5 is the single most misused part of the protocol. Suppression is not a side effect you get by accident; it is a decision encoded in a return value. A method that returns self or a truthy status object from __exit__ will silently swallow every exception in the block.

One more rule from the language reference: __exit__ should not re-raise the exception it was handed. Propagation is the caller's job. Re-raising inside __exit__ corrupts the traceback and makes the failure harder to read, not easier.

The contract in one line: after successful entry, __exit__ fires once per exit path, and its return value is the only switch that decides whether the exception survives.

Watching the Protocol Execute

Before trusting any of this, instrument it. A manager that logs entry, body, and exit events turns the protocol from a claim into an observation.

class Traced:
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        print(f"enter {self.name}")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print(f"exit {self.name} exc_type={exc_type.__name__ if exc_type else None}")
        return False

Run it three ways and read the output.

with Traced("clean"):
    print("body clean")
# enter clean
# body clean
# exit clean exc_type=None

try:
    with Traced("raises"):
        raise ValueError("boom")
except ValueError:
    print("caught outside")
# enter raises
# exit raises exc_type=ValueError
# caught outside

Now flip the return value to True and rerun the second case. The exit line still prints, but caught outside never does. The exception vanished because a single boolean changed. That is the whole suppression mechanism, and it is worth seeing once rather than reading about three times.

The third case is the one the anchor invariant is precise about: acquisition failure.

class FailsToEnter:
    def __enter__(self):
        raise RuntimeError("cannot acquire")
    def __exit__(self, *args):
        print("exit should not run")

try:
    with FailsToEnter():
        print("body should not run")
except RuntimeError:
    print("caught acquisition failure")
# caught acquisition failure

__exit__ never prints. The block never runs. If acquisition has multiple steps and a later step fails after an earlier one succeeded, __enter__ itself must clean up the partial state — the with statement will not do it for you. This is the single most common source of leaked resources in hand-written managers.

class FailsToEnter:
    def __enter__(self):
        raise RuntimeError("cannot acquire")
    def __exit__(self, *args):
        print("exit should not run")

try:
    with FailsToEnter():
        print("body should not run")
except RuntimeError:
    print("caught acquisition failure")

The Smallest Correct Context Manager

Before reaching for any decorator, build the class. You should own the mechanism, not the abstraction.

import socket

class Connection:
    def __init__(self, host, port):
        self.host = host
        self.port = port
        self.sock = None

    def __enter__(self):
        self.sock = socket.create_connection((self.host, self.port))
        return self.sock

    def __exit__(self, exc_type, exc_value, traceback):
        if self.sock is not None:
            self.sock.close()
        return False

Three decisions are visible in fifteen lines.

First, acquisition happens in __enter__, not __init__. If the connection attempt fails inside __init__, you get a half-constructed object that never had a context to exit. If it fails inside __enter__, the with statement never enters the block and __exit__ is never called — which is the correct behavior, because there is nothing to clean up.

Second, cleanup is guarded. self.sock is not None means __exit__ is safe to call even if __enter__ failed partway through. This matters in designs where acquisition has multiple steps and a later step can fail after an earlier one succeeded.

Third, __exit__ returns False. Suppression is opt-in. If you want to swallow an exception, write that decision down and justify it in a comment. Do not let it happen because you returned something truthy by reflex.

Compare the same resource managed by hand:

conn = socket.create_connection((host, port))
try:
    conn.sendall(payload)
finally:
    conn.close()

The difference is not brevity. The difference is that the with version cannot be forgotten at a call site, and the cleanup logic lives in one place instead of being re-derived at every use. When the cleanup rule changes — add a timeout, log the close, track pool metrics — you change one method, not forty call sites.

@contextmanager: The Generator Translation, and Where It Leaks

The generator form is a mechanical translation of the class protocol, not a different mechanism.

from contextlib import contextmanager

@contextmanager
def connection(host, port):
    sock = socket.create_connection((host, port))
    try:
        yield sock
    finally:
        sock.close()

The mapping is exact. Code before yield is __enter__. The yielded value is what as binds. Code after yield is __exit__.

The yield expression re-raises the block's exception inside the generator. That is why try/finally around the yield is mandatory rather than stylistic. Without it, an exception in the block propagates through the generator and the sock.close() line never runs.

Suppression works the same way, expressed differently. If the generator catches the exception and returns normally, the exception is suppressed. If it lets the exception propagate out of the generator, it is not. Same rule, different syntax.

The trap that catches people: a @contextmanager generator must yield exactly once. A generator that yields twice raises RuntimeError at runtime, not at definition time. This bites hardest when the generator has conditional branches:

@contextmanager
def maybe_resource(enabled):
    if enabled:
        yield acquire()
    else:
        yield None  # fine — only one yield executes

That version is correct because only one branch runs. This version is not:

@contextmanager
def broken():
    yield 1
    yield 2  # RuntimeError when the block exits

The error surfaces when the with block tries to exit, which is a confusing place to discover a structural bug.

The standard library already ships the companions you will want. contextlib.closing calls close() on an object that does not implement the protocol itself. contextlib.suppress swallows named exception types. contextlib.ExitStack manages a dynamic number of resources. contextlib.nullcontext gives you a no-op manager for optional resources. Do not reimplement these.

My decision rule: reach for @contextmanager when setup and teardown are a linear sequence of statements. Reach for a class when you need state that outlives a single block, reuse across call sites, reentrancy, or multiple entry points.

@contextmanager
def connection(host, port):
    sock = socket.create_connection((host, port))
    yield sock
    sock.close()

Exception Behavior: Suppression, Chaining, and the Silent Bug

This is where context managers cause the most expensive debugging sessions, because the failure mode is silence.

Consider a manager that wraps a network call and "handles" errors:

@contextmanager
def tolerant_request():
    try:
        yield
    except Exception:
        pass  # the bug

A test that should fail on a connection error now passes. The traceback is gone. The only evidence is a test that succeeds for the wrong reason. I have watched this pattern hide a broken retry loop for a week.

The traceback tells you which of three things happened. Propagation shows the original exception and its full stack. Suppression shows nothing — the exception simply is not there. Re-raising inside __exit__ shows a new exception with the original attached as __context__, which is Python's exception chaining.

Chaining is correct when the cleanup failure is the more useful error. It destroys evidence when the cleanup failure is incidental and the original error is the one you need. A close() that throws during unwinding can replace the real exception with a socket error, and you spend an afternoon debugging the wrong layer.

Handling cleanup failure is a policy choice, not a default. Three positions are defensible, and you should pick one deliberately:

Situation	Reasonable policy	What you lose
No body exception, cleanup fails	Propagate the cleanup failure	Nothing — it is the only signal
Body exception active, cleanup fails	Log or chain the cleanup failure, keep the body exception primary	The cleanup failure may be invisible unless logged
Body exception active, cleanup failure threatens correctness	Let the cleanup failure replace the body exception	The original cause, unless you chain it

The code below takes the second position. It is an example, not a universal rule:

def __exit__(self, exc_type, exc_value, traceback):
    try:
        self.sock.close()
    except OSError:
        if exc_type is None:
            raise  # nothing else is wrong; surface the close failure
        # else: the block already failed; do not mask it
    return False

Suppress only exceptions you can name and justify. Never suppress BaseException subclasses like KeyboardInterrupt or SystemExit — those are control flow, not errors, and swallowing them makes a process that refuses to die.

Ownership, Nesting, and Reentrancy

Single-resource correctness is the easy part. The questions that appear when managers compose are harder.

The ownership rule: the context manager that acquires the resource owns the release. A manager that closes something it did not open is a bug waiting for a second caller. If two managers share a connection pool, neither should close the pool; the pool's own manager should.

Nesting order is LIFO — the last entered is the first exited. For locks this is not stylistic. Reversing the order can deadlock. For transaction scopes, exiting in the wrong order can commit before the inner work is finished.

with transaction(db):
    with lock(resource):
        mutate(resource)
    # lock released here, transaction still open
# transaction commits here

Reentrancy is the next trap. A class-based manager that stores mutable state on self breaks if the same instance is entered twice, including recursively. The Indenter pattern — increment a level on enter, decrement on exit — works for nesting only because the state is a counter, not a flag. A manager that sets self.active = True on enter and False on exit will report False while still inside the outer block.

Either make the manager reentrant deliberately, with a counter or a stack, or document that it is not. Silent non-reentrancy is the worst option.

Thread safety is a separate question. A context manager instance is not automatically safe to share across threads. threading.Lock is safe because the underlying lock is; your custom manager is not, unless you make it so. If two threads enter the same instance, they share self, and any state you stored there is now contested.

When the number of resources is only known at runtime, ExitStack is the tool:

from contextlib import ExitStack

with ExitStack() as stack:
    files = [stack.enter_context(open(p)) for p in paths]
    process(files)
# all files closed in reverse order

You register callbacks as you go and the stack unwinds them in reverse on exit. This is the correct answer to "I have a list of resources and I do not know how many."

with transaction(db):
    with lock(resource):
        mutate(resource)

Async Context Managers and the async with Boundary

The async variant is a distinct protocol, not a footnote. async with requires __aenter__ and __aexit__ that return awaitables. The synchronous protocol does not satisfy it, and the failure is a TypeError at the call site, which is at least honest.

@asynccontextmanager is the async counterpart to @contextmanager, and the pre-yield/post-yield mapping is identical.

The real hazard is blocking cleanup. A synchronous close() that performs network I/O inside an async context manager stalls the event loop for every other task. That is a latency bug, not a style issue. If release genuinely awaits, the manager must be async all the way down.

Cancellation deserves its own warning. CancelledError is not an ordinary exception you can suppress to make shutdown look clean. An __aexit__ that swallows it will leave tasks that refuse to cancel, and the symptom is a process that hangs on shutdown rather than one that crashes.

My rule: use async with when acquisition or release genuinely awaits. Otherwise a synchronous context manager inside an async function is fine and simpler. Do not make things async because the surrounding code is.

Applying This to AI and Service Resources

The resources this audience actually manages — model clients, HTTP sessions, database transactions, temporary artifacts — all benefit from the same discipline.

Wrap HTTP and model clients so connection pools release on every path, including the exception path that a retry loop hides. A retry loop that catches exceptions and retries will mask a leaked connection until the pool is exhausted, and then the failure looks like a capacity problem instead of a cleanup problem.

Transaction scopes encode a policy: commit on success, roll back on exception, always close. Make that policy visible in the name. transaction() that commits is fine; transaction() that sometimes commits and sometimes rolls back based on a flag buried in the body is a debugging tax.

Temporary files and directories for intermediate artifacts should pair creation with deletion. Be explicit about whether cleanup runs on failure. Usually you want it to, but not always — when you are debugging a failed run, the artifact is the evidence.

A short review checklist for any context manager you write or inherit:

• Does __exit__ run on every exit path after successful entry, including return and break?
• Does __enter__ clean up partial acquisition if a later step fails?
• Is cleanup idempotent, and does it tolerate a partial acquisition?
• Is suppression deliberate and named, or accidental?
• Is the manager reentrant, or documented as not?
• Is the async variant actually non-blocking?

And when not to use one: one-shot setup with no teardown, teardown that must outlive the block, or logic that needs to branch on the exception in the caller rather than inside the manager. A context manager that hides a decision the caller needs to make is worse than no context manager at all.

Before you write one, write down the invariant it enforces: what is acquired, what is released, on which paths, and whether suppression is allowed. If you cannot state that in one sentence, the manager is not ready.

Then do the thing that actually teaches you the protocol. Take one resource in your current codebase that is managed with try/finally — or worse, with no cleanup at all — and convert it to a context manager. Then deliberately break it. Raise inside the block. Raise inside the cleanup. Enter it twice. Read the traceback each time. The traceback is the contract's receipt, and it will tell you exactly which guarantee you thought you had and did not.