Python Multiprocessing and Process Pools: When Separate Processes Are Worth It

One core is pinned at 100%. Fifteen sit idle. You wrap the loop in a pool, run it again, and the wall clock barely moves — or gets worse. Nothing about the code looks wrong, which is the problem: the cost that decided the outcome is invisible in the source.

The weak model behind that disappointment is "processes are just threads without the GIL." It is not wrong so much as incomplete. A process boundary is three boundaries stacked on top of each other: a serialization boundary, a memory boundary, and a failure boundary. Each one charges rent, and the rent is paid per task, not per program. Every design decision in this article reduces to one question — is the work behind the boundary large enough to pay for crossing it?

The Boundary Is the Cost Model

Threads share an address space. Asyncio shares an address space and a thread. Processes share nothing. That is the whole trade: you get a private interpreter with its own GIL and true parallel execution, and you pay for it by giving up reference semantics. State moves by copy, not by pointer.

Four costs recur, and they are the ones worth instrumenting:

Cost	What triggers it	When it hurts
Process startup	Pool creation, worker recycling	Short-lived pools, high maxtasksperchild churn
Serialization	Every argument and every return value	Large payloads, high task counts
Memory duplication	Each worker's interpreter and heap	Big parent state, many workers, spawn
Failure transport	Exceptions crossing back to the parent	Unpicklable exceptions, worker crashes

The decision axis is amortization. If a task takes 40 milliseconds and crossing the boundary costs 5 milliseconds each way, you have handed 20% of your runtime to the transport layer before any real work happens. If a task takes 4 seconds, that same overhead rounds to noise.

This is why a pool that loses to a plain for loop is almost never a parallelism problem. The parallelism is fine. The tasks are too small to rent the boundary.

A pool is not a faster loop. It is a loop with a shipping department attached. If the packages are tiny, the shipping costs more than the goods.

Threads and asyncio are the cheap neighbors here: they start fast and share memory, which is exactly why they cannot give you a second core for CPU-bound work. I reach for processes only when the work is CPU-bound, the tasks are independent, and each task is long enough to justify the trip.

Start Methods Change What Your Workers Inherit

The start method is not a portability footnote. It decides what state a worker begins life with, and therefore which bugs are even possible.

Start method	Worker begins as	Inherits parent state	Typical failure
fork	Copy of the parent process	Memory, locks, file descriptors, thread state	Deadlocks from inherited locks
spawn	Fresh interpreter, re-imports the module	Nothing but what you pass	Unpicklable or unimportable targets
forkserver	Fresh interpreter via a server process	Little; server is forked early	Server lifecycle surprises

Under fork, a worker inherits whatever the parent had open at fork time — including a lock held by a thread that does not exist in the child. The child waits forever on a lock nobody will release. That class of deadlock simply cannot occur under spawn, because nothing is inherited.

Under spawn, the opposite tax applies. The worker starts a new interpreter and imports your module from scratch. Module-level side effects run again in every worker. Anything the worker needs must be importable and picklable, which is why the if __name__ == "__main__" guard is a requirement rather than a style preference: without it, the re-import would re-execute your pool creation code inside each worker, recursively.

Defaults have moved across versions and differ by platform — fork is no longer the default everywhere, and the exact default depends on your interpreter and OS. Do not hardcode an assumption about which one you got. Read the active context, and if you are writing a library, accept a context parameter from the caller instead of forcing your own. A library that silently picks a start method creates incompatibilities in someone else's application, and that someone will be you in six months.

Pickling Is the Contract You Did Not Write

Arguments and return values cross the boundary by pickle. That makes serialization an API you never documented but are still bound by.

The canonical failure is a function defined where a spawned worker cannot find it:

from multiprocessing import Pool

def f(x):
    return x * x

if __name__ == "__main__":
    with Pool(5) as p:
        p.map(f, [1, 2, 3])

Run that with f defined inside __main__ under spawn, and you get an AttributeError from the worker: it cannot resolve the attribute f on the module it re-imported. Read the error literally — the worker is telling you it has no way to look up the callable you named. The fix is to move f to an importable module, not to add a retry.

The same rule kills closures, lambdas, open sockets, database connections, and thread locks. The instinct is to send behavior across the boundary. The correct move is almost always to send data and let the worker construct its own behavior.

Large payloads are the quieter tax. Pickling a multi-megabyte array to send it, then pickling the result to return it, can cost more than the computation you were trying to parallelize. Two rules follow:

• Pass small, plain, explicit data across the boundary.
• Keep heavy state inside the worker and initialize it once, not per task.

When a payload genuinely cannot be serialized, you have three exits: manager proxies (which add a server process and IPC round-trips per access), shared memory (fast for raw buffers, awkward for structured state), or restructuring the task so the unserializable object never crosses. Each of those is a real cost, not a workaround. Choose it deliberately.

Building the Smallest Useful Pool

Start boring. A pool over a list of independent tasks, wrapped in a context manager so shutdown is deterministic. The example below is self-contained: load_model is a module-level initializer that stores worker-local state, and score consumes that state instead of receiving it per task.

# worker.py
import multiprocessing as mp

_MODEL = None

def load_model(path):
    global _MODEL
    _MODEL = {"path": path, "bias": 0.5}  # stand-in for a real model load

def score(chunk):
    # CPU-heavy work; returns a small summary, not the raw data
    total = sum(x * x for x in chunk)
    return total * _MODEL["bias"]

if __name__ == "__main__":
    ctx = mp.get_context("spawn")
    chunks = [range(i, i + 10_000) for i in range(0, 100_000, 10_000)]

    with ctx.Pool(
        processes=8,
        initializer=load_model,
        initargs=("model.bin",),
    ) as pool:
        results = pool.map(score, chunks, chunksize=4)

    print(sum(results))

Four knobs in that snippet deserve attention.

close() versus terminate(). close() stops accepting new work and lets queued tasks finish. terminate() stops workers immediately and drops in-flight work on the floor. The context manager calls close() then joins. If you need a hard deadline, you need terminate() plus your own cleanup, and you should know that you are abandoning whatever was mid-flight.

map versus imap. map blocks until every result exists and returns them in input order, which means the full result set lives in memory. imap and imap_unordered stream results as they finish, changing both the memory profile and the latency of the first result. If you only need to consume results once, streaming is usually the better default.

chunksize. This is the knob that trades IPC round-trips against load balancing. A larger chunk means fewer trips across the boundary but coarser distribution: one slow chunk stalls a worker while others idle. The default chunking is often wrong for uneven task durations. If your tasks vary in cost, measure with chunksize=1 and with a larger value before picking.

initializer. Expensive per-worker setup belongs here. Load the model, open the connection, build the index once per worker — not once per task. This is the single highest-leverage line in most real pools.

Now the counterexample. Run the same pool over tasks that take microseconds each, and it loses to a plain loop. That is not a bug. It is the boundary cost made visible, and it is the measurement that should decide whether you keep the pool at all.

Sizing, Memory, and the Fork Tax

More workers is not more throughput. Worker count should track available CPU, not task count. Oversubscription adds context switching and memory pressure while adding zero parallelism.

Memory is where this gets sharp. Each worker is a full interpreter with its own heap. Under spawn, a parent holding 2 GB of state can become N × 2 GB of resident memory, because nothing is shared. Under fork, copy-on-write looks free until someone writes — and the parent or any worker writing a page duplicates it. Treat copy-on-write as a latency optimization, not a memory guarantee.

Long-lived workers accumulate state and leaks. maxtasksperchild recycles them after a fixed number of tasks, at the cost of re-running the initializer each time. That is a real trade: you buy bounded memory growth and pay repeated setup. For workers that load a large model, recycling is expensive; for workers that accumulate small leaks, it is cheap insurance.

Amdahl-style reasoning applies directly. The serial fraction — the parent's own work, the boundary overhead, the final reduction — sets the ceiling. If 20% of your runtime is inherently serial, no worker count gets you past a 5× speedup. Measure the ceiling before scaling workers, and watch resident memory and per-worker CPU utilization, not just wall-clock time.

Lifecycle, Shutdown, and the Deadlock Edges

A pool is a resource with a lifecycle: create, submit, drain, join, clean up. Skipping a step leaves orphaned processes or a hung parent.

Workers in a multiprocessing.Pool are daemonic, which means they cannot spawn their own children. Nested pools need a non-daemonic process or a different architecture entirely. If your design has pools inside pools, stop and reconsider the decomposition.

Finalizers inside pool workers are not guaranteed to run to completion on shutdown. Cleanup logic that must release a lock or flush a buffer does not belong in a worker finalizer, because the pool may exit before it finishes — and an unreleased lock inherited by another worker is a deadlock waiting for a trigger.

A worker that dies mid-task — segfault, OOM kill, os._exit — can leave the parent waiting on a result that will never arrive. Know how your pool surfaces a broken worker versus a raised exception, because the recovery strategy differs.

The classic self-inflicted deadlock is subtler: a worker blocks writing to a full result queue while the parent is not draining it. If the parent is busy submitting more work instead of consuming results, the queue fills, the worker blocks, and the parent waits for a worker that will never finish.

Shutdown needs a bounded escalation path, not a single join(). The context manager handles the happy path: close() then join. When you need a deadline, wrap result consumption in a timeout, and if the deadline is exceeded, call terminate() and then join() again to reap the workers. State explicitly what you are abandoning — in-flight tasks and any worker-local state that was mid-write.

with ctx.Pool(processes=8, initializer=load_model, initargs=("model.bin",)) as pool:
    async_result = pool.map_async(score, chunks, chunksize=4)
    try:
        results = async_result.get(timeout=30)
    except mp.TimeoutError:
        pool.terminate()
        pool.join()
        raise

The deadline belongs around result consumption, not around pool creation. Pool creation is cheap relative to task execution, and putting a timeout there just hides startup cost you should be measuring.

How Failures Cross the Process Boundary

An exception raised in a worker is pickled and re-raised in the parent. What you see is a reconstructed traceback, not the original frame stack. The frames from inside the worker are gone; you get the exception type and message, and a traceback that starts at the boundary.

That has two consequences. First, exceptions that cannot be pickled — custom exceptions carrying unpicklable attributes — turn a clean failure into a confusing secondary error about serialization. Second, the parent's traceback will not tell you which task failed, because the task identity was never part of the exception. Log inside the worker with enough context to identify the task.

The control-flow difference matters too. map fails fast and aborts the batch. imap and async results let you decide per task whether one failure kills the run. And a worker crash is not an exception: a task that raised is a value you can handle, while a process that died is an infrastructure event with different recovery.

My rule: return structured failure data for expected per-task errors, and reserve exceptions for genuinely exceptional conditions. If a task can fail in a way you can name, that failure is data, not an exception.

When Processes Are the Wrong Tool

The decision boundary is worth stating plainly, because the default reach for multiprocessing is usually wrong.

• I/O-bound work belongs in asyncio or threads. Processes add serialization and startup cost for no parallelism gain.
• Tiny tasks at high counts lose to the boundary. Batch them or keep them serial.
• Shared mutable state fights the model. Consider threads, shared memory, or a different decomposition first.
• Work that must run remotely or survive machine failure is a job-queue problem, not a local pool problem.

Use processes when the work is CPU-bound, tasks are independent, payloads are small, and per-task runtime is large enough to amortize the boundary. That is a narrow window, and it is exactly the window where processes win decisively.

The Numbers Should Pick the Configuration

Before you add a pool, measure two things: per-task runtime and payload size. If per-task runtime is measured in milliseconds and payloads are large, the boundary will eat your gains. If per-task runtime is measured in seconds and payloads are small, processes will pay for themselves.

The next action is concrete. Take one real CPU-bound workload and instrument it: run it at two or three pool sizes, with two or three chunksize values, and record wall-clock time alongside resident memory and per-worker CPU utilization. Let the numbers pick the configuration instead of intuition. Then, when the work outgrows a single machine, the adjacent question is how these pools compose with async orchestration and remote execution — a different boundary, with a different rent.