Maybe this behaviour might be viewed as a small bug in distutils-package (as pointed out by @DavidW there is this open issue: https://bugs.python.org/issue35893). However, it also shows, that cythonizing/compiling __init__.py isn't very popular and uses some undocumented implementation details which might change in the future, so it could be wiser to refrain from meddling with __init__.py.
But if you must...
When a package is imported explicitly, e.g.
import ctest
or implicitly, e.g.
import ctest.something
The FileFinder will see that a package, and not a module, is imported and will try to load ctest/__init__.py instead of ctest.py (which most likely doesn't exists):
# Check if the module is the name of a directory (and thus a package).
if cache_module in cache:
base_path = _path_join(self.path, tail_module)
for suffix, loader_class in self._loaders:
init_filename = '__init__' + suffix
full_path = _path_join(base_path, init_filename)
if _path_isfile(full_path):
return self._get_spec(loader_class, fullname, full_path, [base_path], target)
Used suffix, loader_class are for loading __init__.so, __init__.py and __init__.pyc in this order (see also this SO-post). This means, __init__.so will be loaded instead of __init__.py if we manage to create one.
While __init__.py is executed, The property __name__ is the name of the package, i.e. ctest in your case, and not __init__ as one might think. Thus, the name of the init-function, Python-interpreter will call when loading the extension __init__.so is PyInit_ctest in your case (and not PyInit___init__ as one might think).
The above explains, why it all works on Linux out-of-the-box. What about Windows?
The loader can only use symbols from a so/dll which aren't hidden. Per default all symbols built with gcc are visible, but not for VisualStudio on Windows - where all symbols are hidden per default (see e.g. this SO-post).
However, the init-function of a C-extension must be visible (and only the init-function) so it can be called with help of the loader - the solution is to export this symbol (i.e. PyInit_ctest) while linking, in your case it is the wrong /EXPORT:PyInit___init__-option for the linker.
The problem can be found in distutils, or more precise in build_ext-class:
def get_export_symbols(self, ext):
"""Return the list of symbols that a shared extension has to
export. This either uses 'ext.export_symbols' or, if it's not
provided, "PyInit_" + module_name. Only relevant on Windows, where
the .pyd file (DLL) must export the module "PyInit_" function.
"""
initfunc_name = "PyInit_" + ext.name.split('.')[-1]
if initfunc_name not in ext.export_symbols:
ext.export_symbols.append(initfunc_name)
return ext.export_symbols
Here, sadly ext.name has __init__ in it.
From here, one possible solution is easy : to override get_export_symbols, i.e. to add the following to your setup.py-file (read on for a even simpler version):
...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
names = ext.name.split('.')
if names[-1] != "__init__":
initfunc_name = "PyInit_" + names[-1]
else:
# take name of the package if it is an __init__-file
initfunc_name = "PyInit_" + names[-2]
if initfunc_name not in ext.export_symbols:
ext.export_symbols.append(initfunc_name)
return ext.export_symbols
# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...
Calling python setup.py build_ext -i should be enough now (because __init__.so will be loaded rather than __init__.py).
However, as @DawidW has pointed out, Cython uses macro PyMODINIT_FUNC, which is defined as
#define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*
with Py_EXPORTED_SYMBOL being marked as visible/exported on Windows:
#define Py_EXPORTED_SYMBOL __declspec(dllexport)
Thus, there is no need to mark the symbol as visible at the command line. Even worse, this is the reason for the warning LNK4197:
__init__.obj : warning LNK4197: export 'PyInit_ctest' specified multiple times;
using first specification
as PyInit_test is marked as __declspec(dllexport) and exported via option /EXPORT: at the same time.
/EXPORT:-option will be skipped by distutils, if export_symbols is empty, we can use even a simpler version of command.build_ext:
...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
pass # return [] also does the job!
# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...
This is even better than the first version, as it also fixes warning LNK4197!
