python 3.x - spacy 2.2.3 FileNotFoundError: [Errno 2] No such file or directory: 'thinc\neural\_custom_kernels.cu' in pyinstaller

Question

I was trying to create a executable file using pyinstaller. I got bellow issue while executing the issue.

 File "test_env2_livemain.py", line 2, in <module>
 File "C:Users
ajesh.dasAppDataLocalContinuumanaconda3envsest_env2libsite-packagesPyInstallerloaderpyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "test_env2_livecontrollermain.py", line 2, in <module>
File "C:Users
ajesh.dasAppDataLocalContinuumanaconda3envsest_env2libsite-packagesPyInstallerloaderpyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "test_env2_livecontrollerinflowjob_controller.py", line 175, in <module>
File "C:Users
ajesh.dasAppDataLocalContinuumanaconda3envsest_env2libsite-packagesPyInstallerloaderpyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packagesspacy\__init__.py", line 10, in <module>
File "C:Users
ajesh.dasAppDataLocalContinuumanaconda3envsest_env2libsite-packagesPyInstallerloaderpyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packageshinc
eural\__init__.py", line 4, in <module>
File "C:Users
ajesh.dasAppDataLocalContinuumanaconda3envsest_env2libsite-packagesPyInstallerloaderpyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packageshinc
eural\_classesmodel.py", line 11, in <module>
File "C:Users
ajesh.dasAppDataLocalContinuumanaconda3envsest_env2libsite-packagesPyInstallerloaderpyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packageshinc
euralrain.py", line 7, in <module>
File "optimizers.pyx", line 14, in init thinc.neural.optimizers
File "ops.pyx", line 24, in init thinc.neural.ops
ImportError: cannot import name _custom_kernels

So I added --hiddenimport thinc.neural._custom_kernels at the time of build executable file. Bellow code for build executable file

pyinstaller main.py --hiddenimport preshed.maps --hiddenimport srsly.msgpack.util --hidden-import="sklearn.utils._cython_blas" --hidden-import="sklearn.neighbors.typedefs" --hidden-import="sklearn.neighbors.quad_tree" --hidden-import="sklearn.tree._utils" --hidden-import cymem.cymem --hidden-import thinc.linalg --hidden-import murmurhash.mrmr --hidden-import cytoolz.utils --hidden-import cytoolz._signatures --hidden-import spacy.strings --hidden-import spacy.morphology --hidden-import spacy.lexeme --hidden-import spacy.tokens --hidden-import spacy.gold --hidden-import spacy.tokens.underscore --hidden-import spacy.parts_of_speech --hidden-import dill --hidden-import spacy.tokens.printers --hidden-import spacy.tokens._retokenize --hidden-import spacy.syntax --hidden-import spacy.syntax.stateclass --hidden-import spacy.syntax.transition_system --hidden-import spacy.syntax.nonproj --hidden-import spacy.syntax.nn_parser --hidden-import spacy.syntax.arc_eager --hidden-import thinc.extra.search --hidden-import spacy.syntax._beam_utils --hidden-import spacy.syntax.ner --hidden-import thinc.neural._classes.difference --hidden-import spacy.vocab --hidden-import spacy.lemmatizer --hidden-import spacy._ml --hidden-import spacy.lang.en --hiddenimport thinc.neural._custom_kernels

But now executable file gives below error

File "site-packageshinc
euralrain.py", line 7, in <module>
File "optimizers.pyx", line 14, in init thinc.neural.optimizers
File "ops.pyx", line 24, in init thinc.neural.ops
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "C:Users
ajesh.dasAppDataLocalContinuumanaconda3envsest_env2libsite-packagesPyInstallerloaderpyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packageshinc
eural\_custom_kernels.py", line 36, in <module>
File "pathlib.py", line 1183, in open
File "pathlib.py", line 1037, in _opener
File "pathlib.py", line 387, in wrapped
FileNotFoundError: [Errno 2] No such file or directory: 'D:\rajesh\python\console\test_env2_live\dist\main\thinc\neural\_custom_kernels.cu'

Not sure why it gives FileNotFoundError error. Can any one help me?

Configuration: Python 3.6.7, conda 4.8.0, spacy v2.2.3, Windows 10

I can successfully run executable file with lower version of spacy(2.0.18). But it gives error in 2.2.3 version.

Thanks in Advance

Answer 1

The FileNotFound error is because PyInstaller isn't packaging thinc properly; thinc needed a hook. I've found that as script containing from spacy import * will work with the hook file below. The command I used was:

pyinstaller test-spacy.py --additional-hooks-dir=.

This worked because I added a hook for spacy (and it's submodules) in the same directory as the test-spacy.py script. Simply copy the below text into a file called hook-spacy.py, that's in the same directory as your script.

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_all

# ----------------------------- SPACY -----------------------------
data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- THINC -----------------------------
data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- CYMEM -----------------------------
data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- PRESHED -----------------------------
data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]
# This hook file is a bit of a hack - really, all of the libraries should be in seperate hook files. (Eg hook-blis.py with the blis part of the hook)

Ask in the comments if you need any more help.