Generators and yield in Python

python generators solve a problem you've likely already bumped into without naming it: building an entire list in memory when you only actually need to process its values one at a time. The python yield keyword is what makes a generator possible, and this article covers exactly how it works, why generators are so memory-efficient, and the patterns (and pitfalls) worth knowing once you start using them for real.

What Is a Generator?

A generator is a special kind of function that produces values lazily — one at a time, on demand — instead of computing and returning an entire collection all at once.

yield: what makes a function a generator

The yield keyword is what transforms an ordinary function into a generator function. Where return hands back a value and immediately ends the function, yield hands back a value and pauses the function's execution, ready to resume exactly where it left off the next time a value is requested.

def count_up_to(n):
    count = 1
    while count <= n:
        yield count
        count += 1

for number in count_up_to(5):
    print(number)
# 1
# 2
# 3
# 4
# 5

Just the presence of yield anywhere in a function's body is enough to make Python treat the entire function as a generator function — calling it behaves completely differently from a normal function, as the next section explains.

How yield Works: Pause, Resume, and State

Calling a generator function doesn't run its body

This is the detail that surprises people most: calling a generator function doesn't execute any of its code immediately. It returns a generator object instead — the function body only starts running once you actually begin pulling values from that object, typically with next() or a for loop.

def count_up_to(n):
    print("Generator started")
    count = 1
    while count <= n:
        yield count
        count += 1

gen = count_up_to(3)
print(gen)   # <generator object count_up_to at 0x...> — nothing printed yet, no "Generator started"

print(next(gen))   # Generator started
                    # 1
print(next(gen))   # 2
print(next(gen))   # 3

Notice "Generator started" only prints on the first call to next(gen) — not when count_up_to(3) was originally called. The function body genuinely hasn't run at all until something actually asks it for a value.

Local variables and position are preserved between calls

Each call to next() resumes the generator exactly where it left off — with all of its local variables intact, exactly as they were at the moment of the previous yield:

print(next(gen))
# StopIteration

Once the generator has yielded everything it's going to (in this case, after 3), calling next() again raises a StopIteration exception — Python's internal signal that the generator is exhausted. A for loop handles this automatically and silently for you, which is why the earlier example didn't need to catch anything explicitly — but calling next() manually means you're responsible for handling StopIteration yourself if you keep calling past the end.

yield vs. return inside a generator

return still works inside a generator function, but it behaves differently than in a regular function: it doesn't hand back a value the way yield does — instead, it terminates the generator entirely, and the next next() call raises StopIteration.

def limited_counter():
    yield 1
    yield 2
    return   # ends the generator here
    yield 3   # this line is unreachable — never runs

gen = limited_counter()
print(next(gen))   # 1
print(next(gen))   # 2
print(next(gen))
# StopIteration

The core distinction, worth restating plainly: yield pauses execution and preserves state, ready to resume. return (or simply reaching the end of the function body) terminates the generator permanently — there's no resuming after that.

Generator Expressions: The Concise Alternative

Syntax

For simpler cases, Python offers a compact, one-line alternative to writing a full generator function — a generator expression, which looks almost identical to a list comprehension but uses parentheses instead of square brackets:

squares_list = [x ** 2 for x in range(5)]      # list comprehension — builds the full list immediately
squares_gen = (x ** 2 for x in range(5))        # generator expression — lazy, one value at a time
Same lazy behavior, more concise form
gen = (x ** 2 for x in range(5))
print(gen)          # <generator object <genexpr> at 0x...>
print(list(gen))     # [0, 1, 4, 9, 16] — values only computed once you actually consume them

Generator expressions are a good fit for simple, single-expression logic — filtering and transforming, essentially the same territory list comprehensions cover, but without building the full result in memory up front. For anything involving multiple steps, conditional branching beyond a simple filter, or maintaining state across iterations, a full generator function (using def and yield) is generally clearer than trying to cram that logic into a single-line expression — the same readability guidance covered in the earlier list comprehensions article applies here too.

Comparing memory behavior directly
import sys

list_version = [x for x in range(10000)]
gen_version = (x for x in range(10000))

print(sys.getsizeof(list_version))   # a large number — the full list exists in memory
print(sys.getsizeof(gen_version))    # a tiny, fixed number — regardless of range size

The list version has to hold all 10,000 values in memory simultaneously. The generator version holds essentially nothing beyond its own internal state — the values are computed one at a time, only as they're actually requested, which is precisely the subject of the next section.

Why Use Generators: Memory Efficiency and Infinite Sequences

A concrete memory comparison

The gap illustrated above only grows more significant as the numbers involved grow larger. A list comprehension building a million items has to allocate memory for all million values, held simultaneously. A generator producing the same million values, one at a time, uses roughly the same small, constant amount of memory whether it's asked to produce ten values or ten million — because it never holds more than the current value (plus whatever minimal state it needs to compute the next one) at any given moment.

Representing infinite sequences

This is where generators do something a finite list fundamentally cannot: represent a sequence with no defined end at all.

def infinite_counter(start=0):
    count = start
    while True:   # deliberately never stops
        yield count
        count += 1

counter = infinite_counter()
print(next(counter))   # 0
print(next(counter))   # 1
print(next(counter))   # 2
# ...could continue forever, without ever running out of memory

Trying to build this as a list — [x for x in itertools.count()] — would simply run forever, consuming more and more memory until the program crashed. A generator sidesteps this entirely, since it only ever computes (and holds onto) one value at a time, regardless of how many values could theoretically be produced.

Practical real-world use cases

  • Processing large files line by line — iterating a file object in Python is itself already generator-like, reading one line at a time rather than loading an entire multi-gigabyte file into memory at once.

  • Paginated API or database results — a generator can transparently fetch the next page of results only when the caller actually asks for more data, rather than eagerly pulling every page up front.

  • Data pipelines chaining multiple generators together — passing the output of one generator directly into another, building a multi-stage processing pipeline where data flows through one item at a time, rather than materializing a full intermediate list at every stage.

def read_large_file(file_path):
    with open(file_path) as file:
        for line in file:
            yield line.strip()

def filter_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

# Chained together — nothing is loaded into memory all at once
lines = read_large_file("app.log")
errors = filter_errors(lines)

for error_line in errors:
    print(error_line)

This pipeline processes a potentially huge log file one line at a time, all the way through both stages, without ever holding the full file's contents in memory simultaneously.

5. Advanced Patterns and Common Pitfalls

yield from: delegating to a sub-generator

yield from simplifies the common case of one generator needing to yield every value produced by another generator (or any iterable), without writing an explicit inner loop:

def inner_generator():
    yield 1
    yield 2
    yield 3

def outer_generator():
    yield from inner_generator()   # delegates directly to inner_generator
    yield 4

print(list(outer_generator()))   # [1, 2, 3, 4]

Without yield from, you'd need to write out the equivalent manually: for value in inner_generator(): yield value — yield from is simply a cleaner, more direct way to express that same delegation.

Pitfall: an exhausted generator can't be restarted

This trips people up regularly: once a generator has been fully consumed (run to completion, or hit return/StopIteration), it's permanently exhausted — you can't rewind or reuse it. You need to create a brand-new generator instance to iterate over the same logic again:

gen = (x for x in range(3))
print(list(gen))   # [0, 1, 2]
print(list(gen))   # [] — already exhausted, nothing left

# The fix: create a fresh generator
gen = (x for x in range(3))
print(list(gen))   # [0, 1, 2] — works again, since this is a new instance
Pitfall: PEP 479 and StopIteration inside a generator

This is a more advanced pitfall worth knowing about: in older Python versions, if a StopIteration exception happened to be raised (not just naturally reached) somewhere inside a generator's body — say, from a manual next() call on some other iterator inside the generator — it could accidentally terminate the generator silently, in a way that was genuinely hard to debug. PEP 479 changed this behavior: since Python 3.7, an unexpected StopIteration raised inside a generator's body is automatically converted into a RuntimeError instead, making the problem loud and obvious rather than silently swallowed.

def broken_generator():
    yield 1
    raise StopIteration   # don't do this — raises RuntimeError instead, by design
    yield 2

gen = broken_generator()
next(gen)   # 1
next(gen)
# RuntimeError: generator raised StopIteration

The practical takeaway: never manually raise StopIteration inside a generator to intentionally end it — use a plain return instead, exactly as covered in Section 2. return is the correct, well-defined way to end a generator early; manually raising StopIteration is specifically what PEP 479 exists to catch and flag as an error.

A preview: two-way communication with .send()

Worth a brief mention, though it's a genuinely advanced topic beyond the scope of this article: generators aren't limited to one-way communication (producing values outward). The .send() method lets you send a value into a paused generator, which the generator can receive as the result of the yield expression itself. This capability is the foundation of using generators as lightweight coroutines — a pattern that predates, and partly inspired, Python's modern async/await syntax, covered in a later article in this series on asyncio.

PREVIOUSNEXT LESSON