python - Cannot unpickle Exception subclass

Question

Simplified version of Why is my custom exception unpickle failing.

I am trying to pickle a 'simple' exception subclass. It pickles OK, but when unpickling it falls over:

import pickle

class ABError(Exception):
    def __init__(self, a, b):
        self.a = a
        self.b = b

ab_err = ABError("aaaa", "bbbb")

pickled = pickle.dumps(ab_err)
original = pickle.loads(pickled)  # Fails

Error:

Traceback (most recent call last):
  File "p.py", line 12, in <module>
    original = pickle.loads(pickled)  # Fails
  File "/usr/lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
TypeError: __init__() takes exactly 3 arguments (1 given)

An earlier comment suggested the issue is because the built in Exception class supplies a __setstate_() method. However, it's not clear to me if this is expected behaviour or not - it certainly seems surprising since doing the same thing with a subclass of object works OK.

Answer 1

The BaseException class defines a custom __reduce__ method in exceptions.c, which returns the list of arguments to pass to __init__. Exact code is

if (self->args && self->dict)
    return PyTuple_Pack(3, Py_TYPE(self), self->args, self->dict);
else
    return PyTuple_Pack(2, Py_TYPE(self), self->args);

According to __reduce__ documentation,

• the first item of the tuple is the callable to invoke. Here, that will be the exception class.
• the second item is the tuple of arguments to pass to the callable. Here, that will be self.args.
• the third item is a dict to merge into self.__dict__.

So from this, BaseException.__reduce__ will make unpickle invoke the exception's constructor with given args.

You have two options: either override __reduce__, or put the required arguments in self.args, either directly or by letting the parent class do it:

import pickle

class ABError(Exception):
    def __init__(self, a, b):
        self.a = a
        self.b = b
        # self.args = (a, b)
        # maybe better, let base class's __init__ do it =>
        super(ABError, self).__init__(a, b)

ab_err = ABError("aaaa", "bbbb")

pickled = pickle.dumps(ab_err)
original = pickle.loads(pickled)  # no longer fails

Note that the original issue comes from the rather naive way BaseException pickle handling works. It is fixed in the latest python3 releases. Your question's original code works fine on python 3.5 for instance.