Python Iterators, Generators, and Lazy Evaluation: How Iteration Really Works

A generator is not a lazy list. It is a suspended function with a position, and once you can see that position, most "why did my pipeline eat 8 GB" bugs stop being mysterious.

Here is the failure I want you to hold in your head. You write a streaming pipeline over a large dataset. You are careful. You never build the full list. You chain generator stages: read, parse, filter, transform. Then somewhere downstream, a helper calls list() on the stream to compute a count, and a second pass over the same generator returns nothing. No exception. No warning. Just an empty loop and a downstream traceback that points at the consumer instead of the producer.

Two things went wrong, and they are different things. The first is that a generator is single-use: it carries its own position, and once that position reaches the end, it stays there. The second is that laziness is a property of when work happens, not of the object's type. A generator can be forced eager by one list() call. A list can be consumed lazily by iter().

The invariant I want you to leave with: every iterator object carries its own traversal state, and every lazy stage defers work until something pulls. Once you can predict who pulls, when, and how many times, you can predict execution timing, memory behavior, and side-effect ordering. That is the whole game.

The Iterator Protocol Is a State Machine

A for loop is not syntax. It is a small protocol with two methods and one exception.

• iter(obj) calls obj.__iter__() and returns an iterator.
• next(it) calls it.__next__(), which returns the next value or raises StopIteration.
• StopIteration is the normal termination signal. It is not an error.

That is the entire contract. Everything else — comprehensions, for, in, unpacking, sum(), zip() — is built on top of it.

Here is a for loop desugared:

# for x in data: print(x)

_it = iter(data)
while True:
    try:
        x = next(_it)
    except StopIteration:
        break
    print(x)

Read that carefully. The loop does not know anything about data. It only knows how to call iter() once and next() repeatedly. If data is a list, iter(data) returns a fresh list iterator each time. If data is already an iterator, iter(data) returns the same object — because an iterator's __iter__ returns self.

That single design decision is the source of most iterator confusion. Let me make it explicit:

Object	__iter__ returns	Re-iterable?	Iterator holds position?
list, tuple, dict, set	a new iterator each call	yes	no — the container is not the iterator
generator object	itself	no	yes
file object	itself	no	yes
map/filter/zip result	itself	no	yes

The reason containers cannot hold the loop state themselves is concurrency. A list may be iterated by two loops at once, or by a loop inside a function called from another loop. If the list stored "where we are," those loops would fight over one cursor. So the container produces a separate iterator per traversal, and the iterator owns the position.

This is why iter() on a list is cheap and safe, and iter() on a generator is a no-op that hands you back the same exhausted object.

StopIteration Is a Signal, Not a Bug

StopIteration is how an iterator says "I am done." The for loop catches it and exits. But StopIteration is also an exception, and exceptions propagate. That creates a subtle trap inside generators.

Before PEP 479, if a StopIteration escaped from inside a generator body — say, because you called next() on an exhausted iterator without catching it — it would silently terminate the outer generator. The consumer would see a clean end-of-stream and never know a bug had occurred. PEP 479 changed this: a StopIteration that propagates out of a generator frame is now converted into a RuntimeError. That is a deliberate trade. You lose the silent success and gain a loud failure that points at the actual bug.

If you write code that calls next(it, default) inside a generator, you are safe. If you call bare next(it) inside a generator and the inner iterator is exhausted, you will get a RuntimeError at the outer consumer, not a silent truncation. That is the behavior you want.

Generators: Suspended Frames, Not Lazy Lists

The dominant misconception is that a generator is a list that computes its elements on demand. It is not. A generator is a function frame that has been paused mid-execution, and the frame is preserved until you resume it.

Watch the timing:

def stage(name, items):
    print(f"start {name}")
    for item in items:
        print(f"yield {name}: {item}")
        yield item
    print(f"end {name}")

g = stage("A", [1, 2, 3])
print("created")
print(next(g))
print(next(g))

Output:

created
start A
yield A: 1
1
yield A: 2
2

Notice what did not happen. Calling stage("A", [1, 2, 3]) printed nothing. It returned a generator object. The body did not run until the first next(). The print("start A") fired on the first pull, not at definition time.

That is the mechanism: yield suspends the frame, saves the local variables, and returns control to the caller. The next next() resumes the frame at the line after the yield, with all locals intact. This is why a generator can hold a loop counter, an open file handle, or a partially parsed record across calls — the frame is the state.

Generator Expressions vs List Comprehensions

Same syntax shape, opposite execution timing:

squares_list = [x * x for x in range(10)]   # builds all 10 now
squares_gen  = (x * x for x in range(10))   # builds nothing yet

print(squares_list)   # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
print(squares_gen)    # <generator object <genexpr> at 0x...>

The list comprehension allocates a list of 10 ints immediately. The generator expression allocates a generator object and computes each square on demand. For 10 items, the difference is noise. For 10 million items, it is the difference between a working process and an OOM kill.

I default to the generator expression when the result is consumed once and the input is large or unbounded. I default to the list comprehension when I need len(), indexing, or multiple passes, or when the data is small enough that the memory is irrelevant. The syntax is nearly identical; the decision is about consumption, not style.

def mark(x):
    print(f"mark {x}")
    return x * 2

g = (mark(x) for x in [1, 2])
print("built")
print(next(g))

yield from and Composition

yield from delegates iteration to a sub-iterable:

def flatten(nested):
    for group in nested:
        yield from group

It is equivalent to for item in group: yield item, but it also forwards send(), throw(), and close() to the sub-generator. For plain pipelines, the practical benefit is readability: it makes the delegation explicit and avoids a nested loop.

Generators Are Iterators, But Not All Iterators Are Generators

A generator object satisfies the iterator protocol — it has __iter__ and __next__. But you can also write an iterator as a class:

class Countdown:
    def __init__(self, n):
        self.n = n
    def __iter__(self):
        return self
    def __next__(self):
        if self.n <= 0:
            raise StopIteration
        self.n -= 1
        return self.n + 1

Class-based iterators earn their place when you need explicit, inspectable state, or when you want to expose a re-iterable object whose __iter__ returns a fresh iterator each time. A generator is simpler for one-shot streams; a class is clearer when the state itself is the point.

When Laziness Actually Happens

A chain of generator functions is inert until something pulls. This is the single most useful fact for predicting execution timing.

def read(rows):
    for row in rows:
        print(f"read {row}")
        yield row

def parse(rows):
    for row in rows:
        print(f"parse {row}")
        yield row.upper()

pipeline = parse(read(["a", "b", "c"]))
print("built pipeline")
for value in pipeline:
    print(f"got {value}")

Output:

built pipeline
read a
parse a
got A
read b
parse b
got B
read c
parse c
got C

Read the interleaving. read a fires, then parse a, then got A, then read b. The pipeline is not "read everything, then parse everything." It is a pull chain: the consumer asks for one value, which pulls one value from parse, which pulls one value from read, which pulls one row from the source. One item flows through the entire chain before the next item starts.

That is the memory profile of the stages shown above: at any moment, each pull-based generator stage holds one item, not one list. That default breaks the moment a stage buffers — explicit batching, itertools.tee, an eager helper, or a library that prefetches internally. The one-item picture is the anchor; the boundary test is whether any stage in the chain accumulates.

Side Effects Fire at Pull Time

The print calls above are side effects, and they fired during iteration, not at definition. This is where logging gets surprising. If you put a log line inside a generator and never iterate it, the log never appears. If you iterate it twice, the log appears twice — assuming the generator is re-created.

The rule: side effects inside a generator are tied to pulls, not to construction. If you need a side effect to happen at definition time, it does not belong inside the generator body.

Eager Boundaries

Some operations force the entire stream. These are the boundaries where laziness ends:

Operation	Behavior
list(gen)	consumes everything into a list
tuple(gen)	consumes everything into a tuple
sorted(gen)	consumes everything, then sorts
sum(gen), max(gen), min(gen)	consumes everything
any(gen), all(gen)	short-circuits — stops at the first decisive value
set(gen)	consumes everything into a set
"".join(gen)	consumes everything into a string

any() and all() are the interesting ones. They are lazy in the sense that they stop pulling as soon as the answer is determined. any() stops at the first truthy value; all() stops at the first falsy value. Everything else in that table is a full drain.

When you see a memory spike in a lazy pipeline, look for one of these calls. It is almost always a list(), a sorted(), or a join() that someone added to "just check something."

# current code
result = list(records)
found = any(result)

# choose a replacement

Infinite Generators Are Safe Only With Early Termination

An infinite generator is fine as long as a downstream consumer stops:

def naturals():
    n = 0
    while True:
        yield n
        n += 1

from itertools import islice
print(list(islice(naturals(), 5)))   # [0, 1, 2, 3, 4]

islice pulls five values and stops. The generator is suspended, not exhausted. If you replace islice with list(naturals()), the program hangs and then dies on memory. The generator is not the problem; the consumer is.

Building a Lazy Pipeline

The shape I reach for most often is a chain of small generator stages, each pulling one item from the previous one.

import json
from itertools import islice

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip("\n")

def parse_json(lines):
    for line in lines:
        yield json.loads(line)

def keep_valid(records):
    for record in records:
        if record.get("status") == "ok":
            yield record

def batch(iterable, size):
    it = iter(iterable)
    while True:
        chunk = list(islice(it, size))
        if not chunk:
            return
        yield chunk

pipeline = batch(keep_valid(parse_json(read_lines("events.jsonl"))), 100)
for chunk in pipeline:
    process(chunk)

Each stage pulls one item at a time. batch is the only stage that holds more than one item, and it holds exactly size items. The file handle is opened inside read_lines and stays open for the lifetime of that generator — which is a resource-management detail we will come back to.

Where itertools Earns Its Place

itertools is worth knowing because its functions are lazy and composable in ways that hand-written loops often are not:

• chain(a, b) — concatenate iterables without materializing either.
• islice(it, n) — take the first n items, lazily.
• takewhile(pred, it) — take items while a predicate holds, then stop.
• groupby(it, key) — group consecutive items by key. Note: it groups consecutive runs, not all items with the same key. Sort first if you need global grouping, and remember that sorted() is an eager boundary.
• tee(it, n) — split one iterator into n independent iterators, buffering items as needed.

I reach for a hand-written generator when the logic is a one-off transformation with clear control flow. I reach for itertools when the operation is a standard shape — slicing, chaining, grouping — and I want the intent to be obvious at a glance.

The Same Shape Applies to Model Workflows

If you are streaming tokens or responses from a model API, the pipeline shape is identical: a source stage that yields chunks, a parse stage that extracts text or structured fields, a filter stage that drops empty or malformed chunks, and a batch stage that groups them for downstream processing. The laziness matters because model responses can be long, and you often want to start processing the first tokens before the last ones arrive. The same rules apply: one pull at a time, side effects at pull time, and eager boundaries only where you deliberately place them.

The Exhausted Iterator Trap

This is the failure I opened with, and it deserves its own section because it is silent.

def records():
    yield {"id": 1}
    yield {"id": 2}
    yield {"id": 3}

stream = records()

count = sum(1 for _ in stream)   # consumes the generator
print(f"count = {count}")        # count = 3

for record in stream:            # second pass
    print(record)                # prints nothing

No error. No warning. The second loop iterates zero times because the generator is already exhausted. The for loop calls iter(stream), which returns the same exhausted object, and the first next() raises StopIteration, which the loop treats as normal termination.

This is expensive to debug because the failure surfaces downstream, far from the cause. The count pass looks innocent. The empty loop looks like a data problem. The traceback, if there is one, points at whatever consumed the empty result.

Fixes and Their Tradeoffs

Fix	Behavior	Cost
Re-create the generator	call records() again	requires the source to be re-runnable
Wrap in a re-iterable class	__iter__ returns a fresh iterator	more code, but explicit
itertools.tee(stream, 2)	two independent iterators	buffers items not yet consumed by both
Materialize with list()	store everything	memory proportional to the stream

itertools.tee is the tempting one, and it is the one I use least. It works by buffering: when one branch advances ahead of the other, the items it passed are held in memory until the slower branch catches up. If one branch drains the whole stream and the other never runs, tee has effectively materialized the entire stream. It is not free laziness; it is deferred memory.

My rule: if a value must be traversed more than once, it is not a stream. Decide explicitly whether to re-generate it from the source or to store it. Do not paper over the decision with tee and hope the branches stay in sync.

Accidental Eager Work and Other Failure Modes

Beyond the exhausted iterator, there are a few more ways laziness breaks in real code.

Hidden Materialization

A helper function that looks lazy may not be:

def top_n(items, n):
    return sorted(items)[:n]   # sorted() drains the stream

sorted() must see every item before it can return the smallest. If items is a generator over a large dataset, this call materializes the whole thing. The fix depends on the goal: if you need the top n of a stream, use heapq.nlargest(n, items), which keeps only n items in memory. If you need a full sort, you have already decided to pay the memory cost — make that decision visible in the code.

The same applies to library functions. A function that accepts an iterable and internally calls list() on it will drain your stream. Read the signature, not just the name.

Debugging Points at the Consumer

When a generator raises, the traceback shows the consumer's next() call, not the line inside the generator where the exception originated. This is because the generator frame is suspended; the exception propagates out through the next() call.

The practical fix is targeted logging inside the generator, or a wrapper that catches and re-raises with context:

def traced(gen, name):
    try:
        for item in gen:
            yield item
    except Exception as e:
        raise RuntimeError(f"failure in stage {name}") from e

The from e preserves the original traceback while adding the stage name. This is worth the small overhead when you are debugging a multi-stage pipeline.

Resource Management

A generator that opens a file, socket, or database cursor holds that resource open until the generator is exhausted or closed. If the consumer stops early — a break, an exception, a return — the generator is not automatically closed unless it is garbage collected.

The reliable pattern is try/finally inside the generator:

def read_records(path):
    f = open(path)
    try:
        for line in f:
            yield line
    finally:
        f.close()

When the generator is closed — via .close(), garbage collection, or an exception propagating through it — the finally block runs. This is the same mechanism that makes with blocks work inside generators.

The timing is worth being precise about, because it is easy to get backwards. Constructing the generator runs no body code at all. The with open(path) as f: line executes on the first pull, when execution reaches it. The file stays open while the body yields, and the context exits when execution leaves the block — normally after iteration completes, or during generator close or exception unwinding. So the resource lifetime is tied to the generator's execution, not to its construction and not to the moment the with line is written.

def read_records(path):
    with open(path) as f:
        for line in f:
            yield line

gen = read_records("data.txt")   # file not opened yet
first = next(gen)                # file opens here, stays open
gen.close()                      # with block exits, file closes

If the consumer breaks out of the loop early, call gen.close() explicitly — or use contextlib.closing — to release the file deterministically instead of waiting for garbage collection.

When Not to Use Generators

Generators are not free. They add a frame, a suspension point, and a debugging surface. Skip them when:

• The data is small and already in memory. A list is simpler.
• You need len(), indexing, or multiple passes. A list or a re-iterable class is clearer.
• The laziness adds debugging cost without a memory win. If the stream is 100 items, the generator is ceremony.
• The consumer needs to know the size upfront. Generators do not have a length.

The decision is about consumption, not about style. If the value is consumed once and the source is large or unbounded, a generator earns its place. If it is consumed many times or fits comfortably in memory, a list is the honest choice.

A Debugging Checklist for Lazy Pipelines

When a lazy pipeline misbehaves, the failure is almost always one of five things. Walk this list before you start adding print statements at random.

1. Identify the source. Is it re-runnable? A file, a database query, and a generator function are all re-runnable. A network stream or a one-shot API response is not.
2. Mark every consumer. Count how many places iterate the value. If more than one, you have a decision to make before you write the code.
3. Locate the materializers. Search for list(), sorted(), set(), tuple(), join(), and any helper that might call them internally. Each one is an eager boundary.
4. Determine whether the value is re-iterable. If it is a generator, a map/filter/zip result, or a file object, it is not. If it is a list, tuple, dict, or set, it is.
5. Check early termination. If the consumer stops before exhaustion, does the generator close its resources? If not, add try/finally or call .close() explicitly.

That checklist turns a silent exhaustion bug into a five-minute diagnosis instead of an afternoon of guessing.

A Drill You Can Run Today

The fastest way to internalize this is to watch the pull chain with your own eyes.

Write a three-stage pipeline over a synthetic stream, with a print in each stage:

def source():
    for i in range(3):
        print(f"source {i}")
        yield i

def double(items):
    for item in items:
        print(f"double {item}")
        yield item * 2

def label(items):
    for item in items:
        print(f"label {item}")
        yield f"v={item}"

pipeline = label(double(source()))
print("built")
for value in pipeline:
    print(f"got {value}")

Before you run it, predict the output order. Then run it. The interleaving should match the pull chain: source 0, double 0, label 0, got v=0, then source 1, and so on.

Now introduce the bug. Add a count pass before the main loop:

pipeline = label(double(source()))
count = sum(1 for _ in pipeline)
print(f"count = {count}")
for value in pipeline:
    print(f"got {value}")

The second loop prints nothing. The count pass drained the pipeline. Fix it two ways: re-create the pipeline before the loop, and separately, try itertools.tee. Compare the memory behavior. The re-created version re-runs the source; the tee version buffers. Neither is free.

Finally, extend the source to an infinite generator and use islice to take the first five values. Confirm that the program terminates and that the generator is suspended, not exhausted, by checking that a second islice on the same generator continues from where the first stopped.

The Decision Rule

Treat every iterator as a stream with a position. Before you write the pipeline, answer two questions: is this value consumed once or many times, and where do I want the eager boundary to sit? If it is consumed once, a generator is the right shape. If it is consumed many times, it is not a stream — decide explicitly whether to re-generate it or to store it. Place list(), sorted(), and join() deliberately, not by accident.

The next concept in this line is async iteration. async for and async def with yield extend the same protocol to concurrent I/O: the iterator's __anext__ is a coroutine, and the consumer awaits each pull. The state machine is the same; the suspension point now yields to the event loop instead of the caller. If you can predict when a synchronous generator runs, you already have the mental model for when an async generator runs — and for why an unawaited async generator does nothing at all.

def read(path):
    with open(path) as f:
        for line in f:
            yield line

g = read("data.txt")
# no next() has been called yet
first = next(g)
g.close()