What's up with th@? Adding an Operator to CPython

01:15 PM - 01:40 PM on August 16, 2015, Room 702

John Hergenroeder

Audience level:

intermediate

Watch:

http://youtu.be/dQF1hBuQYZE

Description

How do you add an operator to CPython? We'll use the recent addition of the '@' operator in 3.5a1 to answer that question and dig into the CPython internals in the process, touching on how the interpreter goes from source to execution.

Abstract

We'll start by considering our goal: adding a new operator to the CPython implementation of Python. As a case study, we'll use the new infix matrix multiplication operator, '@', which debuted in the release of 3.5a1. In the process, we'll dig into how CPython goes from source code to execution and touch on the tools that the standard library provides for investigating the compilation and interpretation process. Our investigation will also illustrate what it takes to experiment with your own changes to the Python language and cPython.

Adding a new operator requires a broad understanding of how the CPython interpreter runs Python code, going from source code text to tokens, to a parse tree, to an abstract syntax tree (AST), to bytecode, and finally to execution. We'll talk about each of those steps briefly before looking at the changeset that was merged for the operator's inclusion in 3.5.

While the changeset is substantial, we can separate it into four broad changes: updating the grammar used for lexing and parsing, adding a new node type for the AST, extending the internal type object to support matrix multiplication operation definitions, and finally defining and implementing the new bytecodes the interpreter will execute. After walking through those changes, we'll take a moment to note some of the neat tools you can use to investigate CPython on your own.