Unravelling the `with` statement

As part of my series on Python's syntactic sugar, I want to tackle unravelling the with statement.

Looking at the bytecode for a simple with statement, you will notice there are a lot of opcodes being used. Most of it is managing the execution stack for the interpreter and dealing with potential exceptions. As such, I'm going to skip covering the bytecode and instead go with the unravelled syntax as provided by the language reference (touched up for easier reading):

# with a as b:
#     c

_enter = type(a).__enter__
_exit = type(a).__exit__
b = _enter(a)

try:
    c
except:
    if not _exit(a, *sys.exc_info()):
        raise
else:
    _exit(a, None, None, None)

To help illustrate how this all works, we are going to use the classic RAII lock example throughout this blog post. If you're unfamiliar with RAII, it stands for "resource acquisition is initialization". It basically means that by creating an object something happens, and by freeing/deleting the object that something is undone. For locks it means allocating the lock upon creation and then releasing the lock upon deletion. Because Python as a language has no guaranteed cleanup semantics in terms of time and order, context managers were introduced to explicitly bring a language construct to Python that lets us get the benefits of the RAII concept.

import threading

lock = threading.Lock()

with lock:  # The lock is held.
    pass  # Something needing the lock.
# The lock has been released.

There are two parts to a context manager. One is calling its __enter__ special method when entering the with block; for our locking example, this is calling threading.Lock.acquire(). If you use the as clause of a context manager, the object the __enter__ method returns is what gets bound to the variable you specified.

The second part of a context manager is the __exit__ special method when the with block is exited; this is calling threading.Lock.release() in our locking example. The key detail about __exit__ involves whether an exception was raised in the body of the with block. If an exception was raised, it is passed in by its constituent parts as returned by *sys.exc_info() to __exit__; if no exception is raised then None is given for those three parts instead. If an exception happens to be raised in the body of the context manager, the __exit__ method can choose to either suppress or allow the exception to continue to propagate. Which action is taken is controlled by whether a true or false value is returned by the method, respectively. The meaning of the return value is this way so that exceptions will be re-raised thanks to the fact that methods by default return None.

If you would like more historical details, PEP 343 is what brought context managers to Python.