Python Yield, Generators, and Generator Expressions

2020-07-25 09:18:01 | #programming #python | Part 6 of 7

After going through the prerequisites, you should be familiar with iterators such as list, tuples, dicts, and sets. A generator is another type of iterator, but one with a different behavior that really makes it stand out from other iterators.

To go into further detail, iterators utilize next to fetch the next value of a sequence. Generators utilize yield to produce a sequence of values. For a better understand generators, it helps to review some fundamental principles regarding functions.

Whenever a function is called in Python, the program will execute, line-by-line, everything within that function's scope until a return statement or exception is reached. In Python, every function will either return what you've explicitly indicated or None, at which point, any local variables within the function get cleaned up. Unlike iterables, generators generate values as they go and do not store all the values in memory.

With generators, there is no list to be returned. Generators just step through each iteration, and return each value, one by one. Generators usually run faster than regular iterations with the same end result. You also save a lot of processing because you don't have to incur any computations beyond what you've specified in your call to the generator.

Python Yield and Next

To define a generator function, you must utilize yield. As long as yield exists somewhere inside your function, calling it will return a generator.

def generator():
    yield


gen = generator()
print(gen)

yield is a keyword, just like return, however, yield will return a generator at the point it was declared, and continue through the iteration where it left off during the next step. So whereas a return statement permanently hands over control to the caller of the function at the end, yield does so, temporarily, as it goes. The benefit of this is no longer having to keep track of state between calls or having to return large in-memory values at the end of a function call. You return values at each step of the iteration or by calling next() on the generator. This is best illustrated with some examples:

A Recap of Python Iterables

Before we show some generator examples, we should review how to produce the same values with iterables:

Try It Yourself


 
  

Python Generator Example Code

Try It Yourself


 
  

Giving More Control of Returned Values to the Caller

In the above example, we're just printing each value that gets returned with the generator, but of course you can pass these values into other functions. You also have more flexibility over the yielded values' types. So rather than explicitly defining all of the return types across multiple functions, generators allow you to take a more elegant approach, at the same level of performance. Let's look at this, in action, by comparing iterables and generators again.

In the following example, iterables require a separate function for each type:

Try It Yourself


 
  

A More Elegant Solution Using Generators

Try It Yourself


 
  

Sending Values with Python Generators

By calling send() on a generator, you're able to pass values as you iterate:

Try It Yourself


 
  

Terminating Python Generators

You can also terminate generators by calling close() or throwing exceptions with throw().

# Close
def generator():
    try:
        yield
    except GeneratorExit:
        print("Terminating")


gen = generator()
next(gen)
gen.close()


# Throw exception
def generator():
    try:
        yield
    except RuntimeError:
        yield 'value'


gen = generator()
next(gen)
val = gen.throw(RuntimeError, "Something went wrong")

Chaining and Delegation with Python 3

Generators may also chain operations using the yield from <iterable>, syntax. Think of this as shorthand for for i in iterable: yield i:.

def generator(n):
    yield from range(n)
    yield from range(n)


print(list(generator(3)))

yield from also has another added benefit over for loops by allowing subgenerators to receive sent values and thrown exceptions directly from the calling scope, and return a final value to the outer generator. This next example revisits some of what we learned about send.

# Generator 1
def counter():
    cnt = 0
    while True:
        next = yield
        if next is None:
            return cnt
        cnt += next


# Generator 2
def store_totals(totals):
    while True:
        cnt = yield from counter()
        totals.append(cnt)


totals = []  # the list we'll pass to the generator
total = store_totals(totals)
next(total)  # get ready to yield

for i in range(5):
    total.send(i)  # send the values to be totaled up
total.send(None)  # and make sure to stop the generator

for i in range(3):
    total.send(i)  # start back up again
total.send(None)  # and finish the second count

print(totals)

As you've learned in previous sections, all it takes to make a function into a generator is the yield keyword. So we have two generators in this example, with the second generator delegating to the first.

Generator Expressions

Generator expressions, on the other hand, are simpler and can return a generator object without yield and without a generator function.

Try It Yourself


 

Python Programming Exercises

Try to solve the following problems, using everything you've learned up to this point. Feel free to share solutions in the comments. Optimize each solution, as much as possible.

  1. Write a Python generator that prints all the odd integers between 1 and 10

    Input: None

    Expected Output: 1, 3, 5, 7, 9

    Try It Yourself


     
            
  2. Write a Python generator that returns 10 random integers between 1 and 100

    Input: None

    Expected Output: 8, 9, 23, 42, 1, 4, 5, 3, 6, 2

    Try It Yourself


     
            
  3. Write a Python generator that returns 10 random integers between 1 and 100, but step through it only 3 times

    Input: None

    Expected Output: 9, 33, 2

    Try It Yourself


     
            
  4. Write a Python generator that reads rows of the following CSV file, one at a time

    Input:

    id,first_name,last_name,dob
    382,John,Doe,1959/05/23
    121,Jane,Doe,1964/03/03
    329,Jack,Doe,1961/07/15
    119,James,Doe,1989/08/15
    99,Joffrey,Doe,286AC

    Expected Output: 9, 33, 2

    def get_record():
        with open('input.csv', 'r', newline='') as data:
            rows = data.readlines()
            headers = [r.strip('\n\r') for r in rows[0].split(',')]
            for row in rows[1:]:
                record = {}
                for i, col in enumerate(row.split(',')):
                    record[headers[i]] = col.rstrip('\n\r')
                yield(record)
    
    
    gen = get_record()
    print(next(gen))
    print(next(gen))
    print(next(gen))
  5. Write a Python generator that reads rows of the following CSV file, and stops once it reaches an empty row

    Input:

    id,first_name,last_name,dob
    382,John,Doe,1959/05/23
    121,Jane,Doe,1964/03/03
    
    329,Jack,Doe,1961/07/15
    119,James,Doe,1989/08/15
    99,Joffrey,Doe,286AC

    Expected Output:

    {'id': '382', 'first_name': 'John', 'last_name': 'Doe', 'dob': '1959/05/23'}
    {'id': '121', 'first_name': 'Jane', 'last_name': 'Doe', 'dob': '1964/03/03'}
    Traceback (most recent call last):
      File "main.py", line XX, in 
        print(next(gen))
    StopIteration

    def get_record():
        with open('input.csv', 'r', newline='') as data:
            rows = data.readlines()
            headers = [r.strip('\n\r') for r in rows[0].split(',')]
            for row in rows[1:]:
                record = {}
                if row == '\n':
                    break
                for i, col in enumerate(row.split(',')):
                    record[headers[i]] = col.rstrip('\n\r')
                yield(record)
    
    
    gen = get_record()
    print(next(gen))
    print(next(gen))
    print(next(gen))
    print(next(gen))

Want To See More Exercises?

View Exercises

Comments

You must log in to comment. Don't have an account? Sign up for free.

Subscribe to comments for this post

Want To Receive More Free Content?

Would you like to receive free resources, tailored to help you reach your IT goals? Get started now, by leaving your email address below. We promise not to spam. You can also sign up for a free account and follow us on and engage with the community. You may opt out at any time.



Tell Us About Your Project









Contact Us

Do you have a specific IT problem that needs solving or just have a general IT question? Use the contact form to get in touch with us and an IT professional will be with you, momentarily.

Hire Us

We offer web development, enterprise software development, QA &amp; testing, google analytics, domains and hosting, databases, security, IT consulting, and other IT-related services.

Free IT Tutorials

Head over to our tutorials section to learn all about working with various IT solutions.

Contact