cythonize - Cython cimport from another directory

Question

For background, I have read the following questions: https://github.com/cython/cython/wiki/PackageHierarchy

https://cython.readthedocs.io/en/latest/src/userguide/sharing_declarations.html#search-paths-for-definition-files

Cython cimport cannot find .pxd module

How do you get cimport to work in Cython?

Error compiling Cython file: pxd not found in package

The issue is this. I used to have a ton of C, C++, and Cython all sitting in one directory and compiling fine. Now, I am splitting code into 2 directories:

1. C/C++/Cython source for a module "cpysim"
2. C/C++/Cython code for another module "dut"

The rationale is that module cpysim will be reused many times, while module dut will change from project to project. The final wrinkle is that one file in module cpysim can only be compiled with reference to a file in module dut, and that is what's giving me problems.

To be clear, everything was compiling fine when it was all in one directory. This is what it looks like now.

<root>
  -- __init__.py
  -- cpysim
    -- __init__.py
    -- __init__.pxd
    -- sim_core.cpp
    -- sim_core.hpp
    -- sim_core.pxd
    ....
    wiretypes.pyx
    <other sources>
  -- dut
    -- wire_names.def
    -- setup_dut.py
    <other sources>

Goal

Compile wiretypes.pyx from the dut directory (from the setup_dut.py sitting in the dut directory).

Try 1

This import is giving me trouble in wiretypes.pyx

from libcpp cimport bool
from sim_core cimport sigtype_t  # <-- this one
....

This is the relevant contents of setup_dut.py

inc_dirs = ['./', '../', '../cpysim/', '../build', '../dSFMT-src-2.2.3', '../build/include']
....
      Extension("wiretypes",
                ["../cpysim/wiretypes.pyx"],
                language="c++",
                libraries=["cpysim", "ethphy"],
                include_dirs=inc_dirs,
                library_dirs=lib_dirs,
                extra_compile_args=compile_args,
                extra_link_args=link_args),
....
ext = cythonize(extensions,
                gdb_debug=True,
                compiler_directives={'language_level': '3'})

setup(ext_modules=ext,
      cmdclass={'build_ext': build_ext},
      include_dirs=[np.get_include()])

Why I think this should work: according to the documentation, specifying the include path to include the sim_core.pxd header should be enough. For example, cimport numpy as np works when you set include_dirs=[np.get_include()], np.get_include() just spits out a path. So, in inc_dirs, I put ../cpysim. When I compile, I get

Error compiling Cython file:
------------------------------------------------------------
...
"""
Cython header defining a net.
"""

from libcpp cimport bool
from sim_core cimport sigtype_t
^
------------------------------------------------------------

/Users/colinww/system-model/cpysim/wiretypes.pyx:8:0: 'sim_core.pxd' not found

Try 2

I thought maybe I need to treat the cpysim directory as a module. So I added __init__.py, and changed the import in wiretypes.pyx to be:

from libcpp cimport bool
cimport cpysim.sim_core as sim_core

Error compiling Cython file:
------------------------------------------------------------
...
"""
Cython header defining a net.
"""

from libcpp cimport bool
cimport cpysim.sim_core as sim_core
       ^
------------------------------------------------------------

/Users/colinww/system-model/cpysim/wiretypes.pyx:8:8: 'cpysim/sim_core.pxd' not found

So now I'm at a loss. I don't understand why my first attempt did not work, I know the include dirs are being passed properly because there are many other headers which are needed that get found and compile correctly.

I think there is some fundamental aspect of how cimport works that I am missing.

Answer 1

It seems as if you would confuse includes with ... includes.

Building a Cython-extension is a two step process:

1. Generating a C-souce-file from pyx-file, using the cythonize-function and paths to necessary pxd-files as include-paths for the Cython-compiler (to be precise cythonize doesn't call the Cython-compiler directly - it happens later on, when setup is executed, but for the sake of this answer we pretend that cythonized = calling Cython-compiler)

2. Generating an so-file (or what ever) from the generated C-file using needed include paths to headers (*.h-files, e.g. numpy's), when setup-function is called.

What happens if you add include_dirs to Extension? Is it used by the Cython-Compiler or by the C-Compiler?

Cython uses include-directories passed to cythonize-function and in your case it is nothing (which results to [.]), i.e. it must be changed to

ext = cythonize(extensions, include_path=[<path1>, <path2>], ...)

However, Cython also uses sys.path to search for pxd files - so setting sys.path could be a workaround (which is a little bit hacky, as every manipulation of sys.path) - in this case the order of includes is: include_directories, sys.path, Cython/Includes (at least in the most current versions).

---

Funnily, if setuptools without explicit call of cythonize is used, then include_dirs are used by both Cython- and C-compilers,i.e.:

from setuptools import setup, Extension

extensions = [Extension("foo", ["foo.pyx"], include_dirs=[<path1>, <path2>])]
setup(name="foo", ext_modules=extensions)

results in both path1 and path2 being used by both, Cython and C-compilers.

However, I'm not sure the above solution for setting include_path can be recomended: It works only because setuptools uses (also see that) deprecated old_build_ext, which sets include_path here:

...
for source in cython_sources:
    ...
    options = CompilationOptions(...
            include_path = includes,  # HERE WE GO!
            ...)
      result = cython_compile(source, options=options,
                                full_module_name=module_name)