OOP in Python, part 14: The Exception class hierarchy
MP 57: How exceptions are implemented in Python, and what they can show us about class hierarchies.
Note: This post is part of a series about OOP in Python. The previous post discussed how abstract base classes are used in the marshmallow serialization library. The next post looks at the class structure in the pathlib module.
In the last post we discussed one library in depth that used abstract base classes to build a class hierarchy. It’s really informative to see how class hierarchies are implemented in well-established real-world projects, so we’ll continue that trend with an examination of a class hierarchy from the Python standard library.
In this post we’ll look at how exceptions are implemented in Python. Exceptions are always interesting to look at in Python, because everyone has run into them. Even if you haven’t yet learned to write exception-handling code you’ve almost certainly written code that generated a traceback, and there’s an exception at the heart of every unintended traceback.
We’ll spend some time looking at how parts of Python are implemented in C, but you don’t need to know anything about C to understand the larger takeaways from this post.
The exception class hierarchy
One of the challenges of discussing Python class hierarchies in the standard library is that much of the standard library is implemented in C. It’s worth taking a peek at one of these hierarchies, though. For one thing, you can get a much clearer sense of how Python itself is written. But also, most relevant to our ongoing discussion about OOP, you can see that class hierarchies aren’t just a Python thing. They exist in all languages that support object-oriented programming. And even in languages like C that don’t directly support OOP, experienced programmers can still build out effective implementations of OOP concepts.
To understand how exceptions are implemented, let’s focus on a particular one: IndexError
. Here’s a short Python program, bad_list.py, that generates an index error:
python_exceptions = [
"SyntaxError",
"NameError",
]
my_exception = python_exceptions[2]
In this example we’re asking for the third item in python_exceptions
instead of the second item, which generates an error:
$ python bad_list.py
...
IndexError: list index out of range
Let’s look at how IndexError
is implemented, and then come back to this code.
How IndexError
is implemented
The main source code for Python’s built-in exceptions is in a file called exceptions.c. This file is almost 4,000 lines long; searching for IndexError
brings us to this snippet:
/*
* IndexError extends LookupError
*/
SimpleExtendsException(PyExc_LookupError, IndexError,
"Sequence index out of range.");
The first three lines here are a comment. IndexError
inherits from, or extends, LookupError
. SimpleExtendsException()
is a C function that defines how one exception inherits from another.
It turns out IndexError
is just a really thin wrapper around LookupError
. All it adds to LookupError
is a more specific error message, Sequence index out of range. We must be on the right track, because that’s the error message we saw in the output of bad_list.py.
Why would people make such a simple child class? Let’s think about usage for a moment. If you know you’re iterating over a list, you might want to catch IndexError
and handle situations where they arise appropriately. But what if you’re writing a function that could receive either a list or a dictionary? In this situation you can iterate over the sequence, and catch a LookupError
instead of the more specific IndexError
. This will catch IndexError
and KeyError
exceptions, because both of those inherit from LookupError
.
How LookupError
is implemented
It turns out LookupError
is another thin wrapper:
/*
* LookupError extends Exception
*/
SimpleExtendsException(PyExc_Exception, LookupError,
"Base class for lookup errors.");
LookupError
in turn inherits from Exception
.
Let’s keep going; Exception
is yet another thin wrapper:
/*
* Exception extends BaseException
*/
SimpleExtendsException(PyExc_BaseException, Exception,
"Common base class for all non-exit exceptions.");
Exception
inherits from BaseException
, and here we’ve come to the end of child classes that act as thin wrappers.
How BaseException
is implemented
The implementation of BaseException
is hundreds of lines of C code. This code implements most of the exception handling behavior you’re probably familiar with if you’ve used try-except blocks to handle errors.
Here’s the first few lines of BaseException
’s implementation, showing that there’s one more layer to the hierarchy:
/*
* BaseException
*/
static PyObject *
BaseException_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
PyBaseExceptionObject *self;
self = (PyBaseExceptionObject *)type->tp_alloc(type, 0);
if (!self)
return NULL;
/* the dict is created on the fly in PyObject_GenericSetAttr */
self->dict = NULL;
self->notes = NULL;
self->traceback = self->cause = self->context = NULL;
self->suppress_context = 0;
...
BaseException
actually consists of a number of functions that implement the behavior we associate with exceptions. This is the start of the BaseException_new()
function, which defines how new exception objects are created. BaseException
relies on PyObject
, a C component that lies at the base of all Python class hierarchies.
You can see that the implementation in C looks much different than a Python class. Because C doesn’t support OOP directly, this code implements OOP concepts at a lower level. It’s a lot more work, but the benefit is a higher level of efficiency.
Even though the implementation looks a lot different in C, the concepts of OOP and inheritance still show through.
Connecting back to Python
If you’ve been poking around in exceptions.c, you might have noticed one more appearance of IndexError
:
static struct static_exception static_exceptions[] = {
#define ITEM(NAME) {&_PyExc_##NAME, #NAME}
...
ITEM(IndexError), // base: LookupError(Exception)
This is the boundary between C and Python. When we work with an IndexError
in Python, this is the code that connects back to that specific error’s implementation in C.
To wrap up this excursion into C, we’ve found that:
IndexError
inherits fromLookupError
;LookupError
inherits fromException
;Exception
inherits fromBaseException
;BaseException
depends on the more primitiveobject
.1
Seeing this hierarchy in Python
Now that we know what the class hierarchy for IndexError
looks like, we should be able to see it in a couple ways. Let’s go back to the original bad_list.py example, and catch the IndexError
that was raised. Once we’ve caught the exception, we can examine it in several different ways.
We’ll first call help()
on the IndexError
object itself:
python_exceptions = [
"SyntaxError",
"NameError",
]
try:
my_exception = python_exceptions[2]
except IndexError as e:
help(e)
raise e
We place the line that caused the error inside the try
block. The except
block catches IndexError
, and assigns it to the variable e
. It’s this variable that we can examine to learn more about IndexError
objects.
We call help()
on the IndexError
object, and then re-raise the exception so the program exits with the same traceback.
Here’s the most relevant part of the help()
output:
Help on IndexError object:
class IndexError(LookupError)
| Sequence index out of range.
|
| Method resolution order:
| IndexError
| LookupError
| Exception
| BaseException
| object
The help()
output for an instance of a class includes the Method resolution order, or mro, of the class. This is a representation of the class hierarchy. If you call a method on the instance e
, this is a list of the places Python will look for that method.
As an example, exception objects have an add_note()
method, defined in BaseException
:
| Methods inherited from BaseException:
| ...
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
This is at the heart of how inheritance works. If you make a call such as e.add_note("Oh no, index out of bounds!")
, Python will first look in IndexError
for the method add_note()
. If it doesn’t find it there, it will look in the LookupError
implementation. If it doesn’t find the requested method anywhere in the classes listed in the method resolution order, Python will raise an AttributeError
because that method doesn’t exist in the hierarchy.
Calling isinstance()
We can use isinstance()
to verify that the exception we caught is an instance of any class in the hierarchy. For example, here’s how we can prove the IndexError
we caught is also an instance of LookupError
:
python_exceptions = [
...
try:
my_exception = python_exceptions[2]
except IndexError as e:
print(isinstance(e, LookupError))
raise e
Here we’re asking Python whether e
is an instance of LookupError
.
It is:
True
Traceback (most recent call last):
...
You can replace LookupError
with any class in the hierarchy: IndexError
, LookupError
, Exception
, BaseException
, or object
. In all of these cases, the output will be the same. If you replace LookupError
with any exception outside the hierarchy, such as SyntaxError
or NameError
, the output will be False
.
Catching LookupError
This also means you can catch any exception in the hierarchy, and IndexError
will be caught. Here’s what it looks like to catch LookupError
instead of IndexError
:
python_exceptions = [
...
try:
my_exception = python_exceptions[2]
except LookupError as e:
print(type(e))
raise e
Here we’ve used LookupError
in the except
block, and we’re printing the type of the exception object that’s caught.
Here’s the output:
<class 'IndexError'>
Traceback (most recent call last):
...
We’ve still caught an IndexError
, because that’s the exception object that Python raised. But an IndexError
is a LookupError
, so the except
block still runs. This error-handling code will catch any exceptions that inherit from LookupError
, and no other exceptions.
Examining the method resolution order explicitly
An instance by itself doesn’t have a method resolution order; that’s a property of a class. But if you have an instance you can get the class, and then you can get the method resolution order.
Here we’ll go back to catching the IndexError
, and then examine its method resolution order:
python_exceptions = [
...
try:
my_exception = python_exceptions[2]
except LookupError as e:
print(type(e).mro())
raise e
Calling type(e)
returns the class that e
is an instance of. Classes have a default mro()
method, which returns the list of classes where Python will look for methods. That’s effectively a representation of the class hierarchy.
Here’s the output:
[<class 'IndexError'>, <class 'LookupError'>, <class 'Exception'>,
<class 'BaseException'>, <class 'object'>]
Traceback (most recent call last):
...
This is the same set of classes we saw in the implementation chain for IndexError
, and in the help()
output for an IndexError
object.
Conclusions
This was a deeper dive into class hierarchies in Python than I ever intended to do when starting this series. But exploring OOP has led us here, and I’ve learned a lot about Python and OOP by following these rabbit holes. I hope you have as well.
Exceptions are an interesting part of Python to focus on, because most people’s first experience of exceptions centers around seeing them appear when they inevitably make mistakes in their code. It’s easy to think exceptions are just a large group of things that can go wrong, with no real structure to them. But reality is far from that; there’s a whole lot of structure in the implementation of Python’s exceptions, and there’s a lot of purpose behind that structure as well.
The next time you see an exception, consider where it might live in the overall exception class hierarchy. And if you find yourself catching a number of different exceptions, ask yourself if you can move up one level in the hierarchy, and catch the exception that all those others inherit from. If you’re handling them all the same way, this approach might work. If you’re handling each one in a unique way, then you’re probably already working at the correct level in the hierarchy.
Resources
You can find the code files from this post in the mostly_python GitHub repository.
I didn’t show this connection as clearly as all the others. But Python’s base
object
class is at the bottom of all Python class hierarchies, and it has to do with thePyObject
element we saw in the C implementation ofBaseException_new()
. ↩