How Do Python Coroutines Work?
04:00 PM - 04:25 PM on August 16, 2015, Room 702A. Jesse Jiryu Davis
- Audience level:
- advanced
- Watch:
- http://youtu.be/idLtMISlgy8
Description
Python 3's new “asyncio” module is an efficient async framework similar to Node. But unlike Node, it emphasizes a modern idiom called "coroutines", rather than callbacks. Coroutines promise the best of two worlds: the efficiency of callbacks, but with a natural and robust coding style similar to synchronous programming.
In barely 30 minutes I live-code a Python 3 async framework. First, I show how an async framework uses non-blocking sockets, callbacks, and an event loop. This version of the framework is very efficient, but callbacks make a mess of the code. Therefore, I implement coroutines using Python generators and two classes called Future and Task, and update my little framework to use coroutines instead of callbacks.
The live-coding demo isn't just a magic trick: watch to see how simply a coroutine-based async framework can be implemented, and gain a deep understanding of this miraculous new programming idiom in the Python 3 standard library.
Abstract
Classical computer science focuses on CPU-bound problems. It is concerned with algorithms that complete computations quickly. But many networked programs spend their time not computing, but waiting for responses from remote machines. These I/O-bound programs present a very different challenge: wait for a huge number of network responses efficiently. The modern answer to this problem is asynchronous I/O, or “async”.
In the early days of async frameworks, programmers put their logic in callback functions that waited for network responses, made network requests, and then scheduled more callbacks to handle the next round of responses. Complex code descended into an unmanageable mess of spaghetti callbacks. Such code defeated traditional exception handling, and it was intimidating to debug.
Coroutines are a newly popular programming style. They are memory-efficient in I/O-bound applications, but they possess all the grace and simplicity callbacks lack. Best of all, we can keep using time-tested techniques for handling exceptions.
To begin to understand how coroutines work, we look at an example of coroutine code in action. It is a Python generator, but what does that mean? We explore the Python interpreter’s generator implementation, how it uses a stack frame and instruction pointer in an unorthodox way to pause and resume the generator at will.
Coroutines build upon generators: they can pause waiting for a network operation, and resume when the operation completes. I show a minimal implementation of the Future and Task classes used in asyncio to schedule coroutines.
Coroutines can be factored into subcoroutines, the same as regular functions can. We see how Python 3’s “yield from” statement allows us to factor coroutines. Most comfortingly, exception-handling works with coroutines the same as with functions. We get a sane stack trace that shows where our program failed.