python - reload module with pyximport?

Question

I have a python program that loads quite a bit of data before running. As such, I'd like to be able to reload code without reloading data. With regular python, importlib.reload has been working fine. Here's an example:

setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

extensions = [
    Extension("foo.bar", ["foo/bar.pyx"],
              language="c++",
              extra_compile_args=["-std=c++11"],
              extra_link_args=["-std=c++11"])
]
setup(
    name="system2",
    ext_modules=cythonize(extensions, compiler_directives={'language_level' : "3"}),
)

foo/bar.py

cpdef say_hello():
    print('Hello!')

runner.py:

import pyximport
pyximport.install(reload_support=True)

import foo.bar
import subprocess
from importlib import reload

if __name__ == '__main__':

    def reload_bar():
        p = subprocess.Popen('python setup.py build_ext --inplace',
                             shell=True,
                             cwd='<your directory>')
        p.wait()

        reload(foo.bar)
        foo.bar.say_hello()

But this doesn't seem to work. If I edit bar.pyx and run reload_bar I don't see my changes. I also tried pyximport.build_module() with no luck -- the module rebuilt but didn't reload. I'm running in a "normal" python shell, not IPython if it makes a difference.

Answer 1

I was able to get a solution working for Python 2.x a lot easier than Python 3.x. For whatever reason, Cython seems to be caching the shareable object (.so) file it imports your module from, and even after rebuilding and deleting the old file while running, it still imports from the old shareable object file. However, this isn't necessary anyways (when you import foo.bar, it doesn't create one), so we can just skip this anyways.

The largest problem was that python kept a reference to the old module, even after reloading. Normal python modules seem to work find, but not anything cython related. To fix this, I run execute two statements in place of reload(foo.bar)

del sys.modules['foo.bar']
import foo.bar

This successfully (though probably less efficiently) reloads the cython module. The only issue that remains in in Python 3.x running that subprocess creates a problematic shareable objects. Instead, skip that all together and let the import foo.bar work its magic with the pyximporter module, and recompile for you. I also added an option to the pyxinstall command to specify the language level to match what you've specified in the setup.py

pyximport.install(reload_support=True, language_level=3)

So all together:

runner.py

import sys
import pyximport
pyximport.install(reload_support=True, language_level=3)

import foo.bar

if __name__ == '__main__':
    def reload_bar():
        del sys.modules['foo.bar']
        import foo.bar

    foo.bar.say_hello()
    input("  press enter to proceed  ")
    reload_bar()
    foo.bar.say_hello()

Other two files remained unchanged

Running:

Hello!
  press enter to proceed

-replace "Hello!" in foo/bar.pyx with "Hello world!", and press Enter.

Hello world!