Python Yield, Generators, and Generator Expressions

2020-07-25 09:18:01 | #programming #python

After going through the prerequisites, you should be familiar with iterators such as list, tuples, dicts, and sets. A generator is another type of iterator, but one with a different behavior that really makes it stand out from other iterators.

To go into further detail, iterators utilize next to fetch the next value of a sequence. Generators utilize yield to produce a sequence of values. For a better understand generators, it helps to review some fundamental principles regarding functions.

Whenever a function is called in Python, the program will execute, line-by-line, everything within that function's scope until a return statement or exception is reached. In Python, every function will either return what you've explicitly indicated or None, at which point, any local variables within the function get cleaned up. Unlike iterables, generators generate values as they go and do not store all the values in memory.

With generators, there is no list to be returned. Generators just step through each iteration, and return each value, one by one. Generators usually run faster than regular iterations with the same end result. You also save a lot of processing because you don't have to incur any computations beyond what you've specified in your call to the generator.

Python Yield and Next

To define a generator function, you must utilize yield. As long as yield exists somewhere inside your function, calling it will return a generator.

def generator():

gen = generator()

yield is a keyword, just like return, however, yield will return a generator at the point it was declared, and continue through the iteration where it left off during the next step. So whereas a return statement permanently hands over control to the caller of the function at the end, yield does so, temporarily, as it goes. The benefit of this is no longer having to keep track of state between calls or having to return large in-memory values at the end of a function call. You return values at each step of the iteration or by calling next() on the generator. This is best illustrated with some examples:

A Recap of Python Iterables

Before we show some generator examples, we should review how to produce the same values with iterables:

Try It Yourself


Python Generator Example Code

Try It Yourself


Giving More Control of Returned Values to the Caller

In the above example, we're just printing each value that gets returned with the generator, but of course you can pass these values into other functions. You also have more flexibility over the yielded values' types. So rather than explicitly defining all of the return types across multiple functions, generators allow you to take a more elegant approach, at the same level of performance. Let's look at this, in action, by comparing iterables and generators again.

In the following example, iterables require a separate function for each type:

Try It Yourself


A More Elegant Solution Using Generators

Try It Yourself


Sending Values with Python Generators

By calling send() on a generator, you're able to pass values as you iterate:

Try It Yourself


Terminating Python Generators

You can also terminate generators by calling close() or throwing exceptions with throw().

# Close
def generator():
    except GeneratorExit:

gen = generator()

# Throw exception
def generator():
    except RuntimeError:
        yield 'value'

gen = generator()
val = gen.throw(RuntimeError, "Something went wrong")

Chaining and Delegation with Python 3

Generators may also chain operations using the yield from <iterable>, syntax. Think of this as shorthand for for i in iterable: yield i:.

def generator(n):
    yield from range(n)
    yield from range(n)


yield from also has another added benefit over for loops by allowing subgenerators to receive sent values and thrown exceptions directly from the calling scope, and return a final value to the outer generator. This next example revisits some of what we learned about send.

# Generator 1
def counter():
    cnt = 0
    while True:
        next = yield
        if next is None:
            return cnt
        cnt += next

# Generator 2
def store_totals(totals):
    while True:
        cnt = yield from counter()

totals = []  # the list we'll pass to the generator
total = store_totals(totals)
next(total)  # get ready to yield

for i in range(5):
    total.send(i)  # send the values to be totaled up
total.send(None)  # and make sure to stop the generator

for i in range(3):
    total.send(i)  # start back up again
total.send(None)  # and finish the second count


As you've learned in previous sections, all it takes to make a function into a generator is the yield keyword. So we have two generators in this example, with the second generator delegating to the first.

Generator Expressions

Generator expressions, on the other hand, are simpler and can return a generator object without yield and without a generator function.

Try It Yourself



You must log in to comment. Don't have an account? Sign up for free.

Subscribe to Our Newsletter

Would you like to receive free whitepapers and other IT news? Just leave your email address below. You may opt out at any time.

Tell Us About Your Project

Contact Us

Do you have a specific IT problem that needs solving or just have a general IT question? Use the contact form to get in touch with us and an IT professional will be with you, momentarily.

Hire Us

We offer web development, enterprise software development, QA &amp; testing, google analytics, domains and hosting, databases, security, IT consulting, and other IT-related services.

Free IT Tutorials

Head over to our tutorials section to learn all about working with various IT solutions.