Python Yield, Generators, and Generator Expressions

Whenever a function is called in Python, the program will execute, line-by-line, everything within that function's scope until a return statement or exception is reached. In Python, every function will either return what you've explicitly indicated or None, at which point, any local variables within the function get cleaned up. Unlike iterables, generators generate values as they go and do not store all the values in memory.

With generators, there is no list to be returned. Generators just step through each iteration, and return each value, one by one. Generators usually run faster than regular iterations with the same end result. You also save a lot of processing because you don't have to incur any computations beyond what you've specified in your call to the generator.

Python Yield and Next

To define a generator function, you must utilize yield. As long as yield exists somewhere inside your function, calling it will return a generator.

def generator():
    yield


gen = generator()
print(gen)

yield is a keyword, just like return, however, yield will return a generator at the point it was declared, and continue through the iteration where it left off during the next step. So whereas a return statement permanently hands over control to the caller of the function at the end, yield does so, temporarily, as it goes. The benefit of this is no longer having to keep track of state between calls or having to return large in-memory values at the end of a function call. You return values at each step of the iteration or by calling next() on the generator. This is best illustrated with some examples:

A Recap of Python Iterables

Before we show some generator examples, we should review how to produce the same values with iterables:

Python Generator Example Code

Giving More Control of Returned Values to the Caller

In the above example, we're just printing each value that gets returned with the generator, but of course you can pass these values into other functions. You also have more flexibility over the yielded values' types. So rather than explicitly defining all of the return types across multiple functions, generators allow you to take a more elegant approach, at the same level of performance. Let's look at this, in action, by comparing iterables and generators again.

In the following example, iterables require a separate function for each type:

A More Elegant Solution Using Generators

Sending Values with Python Generators

By calling send() on a generator, you're able to pass values as you iterate:

Terminating Python Generators

You can also terminate generators by calling close() or throwing exceptions with throw().

# Close
def generator():
    try:
        yield
    except GeneratorExit:
        print("Terminating")


gen = generator()
next(gen)
gen.close()


# Throw exception
def generator():
    try:
        yield
    except RuntimeError:
        yield 'value'


gen = generator()
next(gen)
val = gen.throw(RuntimeError, "Something went wrong")

Chaining and Delegation with Python 3

Generators may also chain operations using the yield from <iterable>, syntax. Think of this as shorthand for for i in iterable: yield i:.

def generator(n):
    yield from range(n)
    yield from range(n)


print(list(generator(3)))

yield from also has another added benefit over for loops by allowing subgenerators to receive sent values and thrown exceptions directly from the calling scope, and return a final value to the outer generator. This next example revisits some of what we learned about send.

# Generator 1
def counter():
    cnt = 0
    while True:
        next = yield
        if next is None:
            return cnt
        cnt += next


# Generator 2
def store_totals(totals):
    while True:
        cnt = yield from counter()
        totals.append(cnt)


totals = []  # the list we'll pass to the generator
total = store_totals(totals)
next(total)  # get ready to yield

for i in range(5):
    total.send(i)  # send the values to be totaled up
total.send(None)  # and make sure to stop the generator

for i in range(3):
    total.send(i)  # start back up again
total.send(None)  # and finish the second count

print(totals)

As you've learned in previous sections, all it takes to make a function into a generator is the yield keyword. So we have two generators in this example, with the second generator delegating to the first.

Generator Expressions

Generator expressions, on the other hand, are simpler and can return a generator object without yield and without a generator function.

Python Programming Exercises

Try to solve the following problems, using everything you've learned up to this point. Feel free to share solutions in the comments. Optimize each solution, as much as possible.

1. Write a Python generator that prints all the odd integers between 1 and 10

   Input: None

   Expected Output: 1, 3, 5, 7, 9

   Show solution

   Try It Yourself

   def get_numbers(): for i in range(1, 11): yield i for i in get_numbers(): print(i)
   Run

2. Write a Python generator that returns 10 random integers between 1 and 100

   Input: None

   Expected Output: 8, 9, 23, 42, 1, 4, 5, 3, 6, 2

   Show solution

   Try It Yourself

   import random def get_numbers(): for i in range(1, 11): yield random.randint(1, 100) for i in get_numbers(): print(i)
   Run

3. Write a Python generator that returns 10 random integers between 1 and 100, but step through it only 3 times

   Input: None

   Expected Output: 9, 33, 2

   Show solution

   Try It Yourself

   import random def get_numbers(): for i in range(1, 11): yield random.randint(1, 100) gen = get_numbers() print(next(gen)) print(next(gen)) print(next(gen))
   Run

4. Write a Python generator that reads rows of the following CSV file, one at a time

   Input:
   

   id,first_name,last_name,dob
   382,John,Doe,1959/05/23
   121,Jane,Doe,1964/03/03
   329,Jack,Doe,1961/07/15
   119,James,Doe,1989/08/15
   99,Joffrey,Doe,286AC

   Expected Output: 9, 33, 2

   Show solution

   def get_record():
       with open('input.csv', 'r', newline='') as data:
           rows = data.readlines()
           headers = [r.strip('\n\r') for r in rows[0].split(',')]
           for row in rows[1:]:
               record = {}
               for i, col in enumerate(row.split(',')):
                   record[headers[i]] = col.rstrip('\n\r')
               yield(record)
   
   
   gen = get_record()
   print(next(gen))
   print(next(gen))
   print(next(gen))

5. Write a Python generator that reads rows of the following CSV file, and stops once it reaches an empty row

   Input:
   

   id,first_name,last_name,dob
   382,John,Doe,1959/05/23
   121,Jane,Doe,1964/03/03
   
   329,Jack,Doe,1961/07/15
   119,James,Doe,1989/08/15
   99,Joffrey,Doe,286AC

   Expected Output:
   

   {'id': '382', 'first_name': 'John', 'last_name': 'Doe', 'dob': '1959/05/23'}
   {'id': '121', 'first_name': 'Jane', 'last_name': 'Doe', 'dob': '1964/03/03'}
   Traceback (most recent call last):
     File "main.py", line XX, in 
       print(next(gen))
   StopIteration

   Show solution

   def get_record():
       with open('input.csv', 'r', newline='') as data:
           rows = data.readlines()
           headers = [r.strip('\n\r') for r in rows[0].split(',')]
           for row in rows[1:]:
               record = {}
               if row == '\n':
                   break
               for i, col in enumerate(row.split(',')):
                   record[headers[i]] = col.rstrip('\n\r')
               yield(record)
   
   
   gen = get_record()
   print(next(gen))
   print(next(gen))
   print(next(gen))
   print(next(gen))

Conclusion

That's the end of this tutorial. We hope you found it helpful. Make sure to check out our other tutorials, as well.