Python Yield, Generators, and Generator Expressions
2020-07-25 09:18:01 | | Part 6 of 7
Tested On
- Linux Ubuntu 20.04
- Windows 10
- macOS Catalina
After going through the prerequisites, you should be familiar with iterators such as list, tuples, dicts, and sets. A generator is another type of iterator, but one with a different behavior that really makes it stand out from other iterators.
To go into further detail, iterators utilize next to fetch the next value of a sequence. Generators utilize yield to produce a sequence of values. For a better understand generators, it helps to review some fundamental principles regarding functions.
Whenever a function is called in Python, the program will execute, line-by-line, everything within that function's scope until a return statement or exception is reached. In Python, every function will either return what you've explicitly indicated or None, at which point, any local variables within the function get cleaned up. Unlike iterables, generators generate values as they go and do not store all the values in memory.
With generators, there is no list to be returned. Generators just step through each iteration, and return each value, one by one. Generators usually run faster than regular iterations with the same end result. You also save a lot of processing because you don't have to incur any computations beyond what you've specified in your call to the generator.
Python Yield and Next
To define a generator function, you must utilize yield. As long as yield exists somewhere inside your function, calling it will return a generator.
def generator():
yield
gen = generator()
print(gen)
yield is a keyword, just like return, however, yield will return a generator at the point it was declared, and continue through the iteration where it left off during the next step. So whereas a return statement permanently hands over control to the caller of the function at the end, yield does so, temporarily, as it goes. The benefit of this is no longer having to keep track of state between calls or having to return large in-memory values at the end of a function call. You return values at each step of the iteration or by calling next() on the generator. This is best illustrated with some examples:
A Recap of Python Iterables
Before we show some generator examples, we should review how to produce the same values with iterables:
Try It Yourself
Python Generator Example Code
Try It Yourself
Giving More Control of Returned Values to the Caller
In the above example, we're just printing each value that gets returned with the generator, but of course you can pass these values into other functions. You also have more flexibility over the yielded values' types. So rather than explicitly defining all of the return types across multiple functions, generators allow you to take a more elegant approach, at the same level of performance. Let's look at this, in action, by comparing iterables and generators again.
In the following example, iterables require a separate function for each type:
Try It Yourself
A More Elegant Solution Using Generators
Try It Yourself
Sending Values with Python Generators
By calling send() on a generator, you're able to pass values as you iterate:
Try It Yourself
Terminating Python Generators
You can also terminate generators by calling close() or throwing exceptions with throw().
# Close
def generator():
try:
yield
except GeneratorExit:
print("Terminating")
gen = generator()
next(gen)
gen.close()
# Throw exception
def generator():
try:
yield
except RuntimeError:
yield 'value'
gen = generator()
next(gen)
val = gen.throw(RuntimeError, "Something went wrong")
Chaining and Delegation with Python 3
Generators may also chain operations using the yield from <iterable>, syntax. Think of this as shorthand for for i in iterable: yield i:.
def generator(n):
yield from range(n)
yield from range(n)
print(list(generator(3)))
yield from also has another added benefit over for loops by allowing subgenerators to receive sent values and thrown exceptions directly from the calling scope, and return a final value to the outer generator. This next example revisits some of what we learned about send.
# Generator 1
def counter():
cnt = 0
while True:
next = yield
if next is None:
return cnt
cnt += next
# Generator 2
def store_totals(totals):
while True:
cnt = yield from counter()
totals.append(cnt)
totals = [] # the list we'll pass to the generator
total = store_totals(totals)
next(total) # get ready to yield
for i in range(5):
total.send(i) # send the values to be totaled up
total.send(None) # and make sure to stop the generator
for i in range(3):
total.send(i) # start back up again
total.send(None) # and finish the second count
print(totals)
As you've learned in previous sections, all it takes to make a function into a generator is the yield keyword. So we have two generators in this example, with the second generator delegating to the first.
Generator Expressions
Generator expressions, on the other hand, are simpler and can return a generator object without yield and without a generator function.
Try It Yourself
Python Programming Exercises
Try to solve the following problems, using everything you've learned up to this point. Feel free to share solutions in the comments. Optimize each solution, as much as possible.
-
Write a Python generator that prints all the odd integers between 1 and 10
Input: None
Expected Output: 1, 3, 5, 7, 9
-
Write a Python generator that returns 10 random integers between 1 and 100
Input: None
Expected Output: 8, 9, 23, 42, 1, 4, 5, 3, 6, 2
-
Write a Python generator that returns 10 random integers between 1 and 100, but step through it only 3 times
Input: None
Expected Output: 9, 33, 2
-
Write a Python generator that reads rows of the following CSV file, one at a time
Input:
id,first_name,last_name,dob 382,John,Doe,1959/05/23 121,Jane,Doe,1964/03/03 329,Jack,Doe,1961/07/15 119,James,Doe,1989/08/15 99,Joffrey,Doe,286AC
Expected Output: 9, 33, 2
-
Write a Python generator that reads rows of the following CSV file, and stops once it reaches an empty row
Input:
id,first_name,last_name,dob 382,John,Doe,1959/05/23 121,Jane,Doe,1964/03/03 329,Jack,Doe,1961/07/15 119,James,Doe,1989/08/15 99,Joffrey,Doe,286AC
Expected Output:
{'id': '382', 'first_name': 'John', 'last_name': 'Doe', 'dob': '1959/05/23'} {'id': '121', 'first_name': 'Jane', 'last_name': 'Doe', 'dob': '1964/03/03'} Traceback (most recent call last): File "main.py", line XX, in
print(next(gen)) StopIteration
Conclusion
That's the end of this tutorial. We hope you found it helpful. Make sure to check out our other tutorials, as well.
Comments
You must log in to comment. Don't have an account? Sign up for free.
Subscribe to comments for this post
Info