python - debugging pickle

Question

I'm trying to pickle quite an involved object hierarchy and getting the exception:

pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed

Are there any reasonable methods one can use to test the pickleablility of an object hierarchy? My aim would be to find the location of the offending function

Answer 1

To do this, I'd use dill, which can serialize almost anything in python. Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

>>> import dill
>>> dill.loads(dill.dumps(your_bad_object))
>>> ...
>>> # if you get a pickling error, use dill's tools to figure out a workaround
>>> dill.detect.badobjects(your_bad_object, depth=0)
>>> dill.detect.badobjects(your_bad_object, depth=1)
>>> ...

If you absolutely wanted to, you could use dill's badobjects (or one of the other detection functions) to dive recursively into your object's reference chain, and pop out the unpickleable objects, instead of calling it at at every depth, as above.

Also, objgraph is a pretty handy compliment to the test suite too.

>>> # visualize the references in your bad objects
>>> objgraph.show_refs(your_bad_object, filename='your_bad_object.png')