python - cx-freeze doesn't find all dependencies

Question

I have a python script (2.7) with some "interesting" imports at the top. I initially wanted to use py2exe to compile this into an exe that I can more easily distribute (py2exe doesn't find all dependencies).

I have given up and am trying to use cx-freeze instead. But, I am having problem there as well. The problems seem to be libraries I have added to Python (jinja2 and restkit). I see them in my python directory ./Lib/site-packages/Jinja2-2.6-py2.7.egg/jinja2 and here ./Lib/site-packages/restkit-4.2.1-py2.7.egg/restkit.

Here are the imports in my script:

import datetime
from jinja2 import Environment, PackageLoader
from optparse import OptionParser
from datetime import date, timedelta
from restkit import Resource, BasicAuth, request

I am using a setup.py with cx-freeze. Here is the setup.py:

from cx_Freeze import setup, Executable


packages  = ["restkit", "jinja2" , "restkit.client" ]
includes = []
includefiles = []
eggsacutibull = Executable(
    script = "myScript.py",
    initScript = None,
    targetName = "myScript.exe",
    compress = True,
    copyDependentFiles = True,
    appendScriptToExe = False,
    appendScriptToLibrary = False,
    icon = None
    )

setup(
        name = "myScript",
        version = "0.1",
        author = 'vickery',
        description = "MyScript description",
        options = {"build_exe": {"includes":includes, "include_files": includefiles, "packages": packages}},
        executables = [eggsacutibull]
        )

I run cxfreeze like this:

cxfreeze myScript.py --target-dir exe

I get this in my build:

Missing modules:
? __pypy__ imported from jinja2.debug
? http_parser.http imported from restkit.client
? jinja2._debugsupport imported from jinja2.debug
? jinja2._markupsafe._speedups imported from jinja2._markupsafe
? jinja2.debugrenderer imported from jinja2.debug
? markupsafe imported from jinja2.utils
? pretty imported from jinja2.utils
? socketpool imported from restkit.conn

And, when I try to run the exe, I get this:

Traceback (most recent call last):
  File "c:Python27libsite-packages
estkit-4.2.1-py2.7.egg
estkit\__init__.py", line 9, in <module>
    from restkit.conn import Connection
  File "c:Python27libsite-packages
estkit-4.2.1-py2.7.egg
estkitconn.py", line 14, in <module>
    from socketpool import Connector
ImportError: No module named socketpool
Traceback (most recent call last):
  File "c:Python27libsite-packagescx_FreezeinitscriptsConsole.py", line 27, in <module>
    exec code in m.__dict__
  File "myScript.py", line 12, in <module>
ImportError: cannot import name Resource

Edit: I am now running cxfreeze correctly like this:

python setup.py build

I also added socketpool to my setup.py:

packages = [ "restkit", "jinja2" , "restkit.client", "restkit.conn", "socketpool" ]

But, when I try to build now, I get a build error:

$ python setup.py build
running build
running build_exe
Traceback (most recent call last):
  File "setup.py", line 32, in <module>
    executables = [eggsacutibull]
  File "c:python27libsite-packagescx_Freezedist.py", line 365, in setup
    distutils.core.setup(**attrs)
  File "c:python27libdistutilscore.py", line 152, in setup
    dist.run_commands()
  File "c:python27libdistutilsdist.py", line 953, in run_commands
    self.run_command(cmd)
  File "c:python27libdistutilsdist.py", line 972, in run_command
    cmd_obj.run()
  File "c:python27libdistutilscommanduild.py", line 127, in run
    self.run_command(cmd_name)
  File "c:python27libdistutilscmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "c:python27libdistutilsdist.py", line 972, in run_command
    cmd_obj.run()
  File "c:python27libsite-packagescx_Freezedist.py", line 235, in run
    freezer.Freeze()
  File "c:python27libsite-packagescx_Freezefreezer.py", line 570, in Freeze
    self.finder = self._GetModuleFinder()
  File "c:python27libsite-packagescx_Freezefreezer.py", line 325, in _GetModuleFinder
    finder.IncludePackage(name)
  File "c:python27libsite-packagescx_Freezefinder.py", line 534, in IncludePackage
    module = self._ImportModule(name, deferredImports)
  File "c:python27libsite-packagescx_Freezefinder.py", line 274, in _ImportModule
    raise ImportError("No module named %r" % name)
ImportError: No module named 'socketpool'

What has me confused here, is, my script compiles just fine. In addition, I can import these modules from a python shell. For example:

$ python
Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import socketpool
>>> from restkit import Resource, BasicAuth, request
>>>

How is python resolving modules that is differnt than cxfreeze?

Edit2: From python I can do this:

>>> import socketpool
>>> print socketpool.__file__
c:python27libsite-packagessocketpool-0.5.2-py2.7.eggsocketpool\__init__.pyc

Is that a non-standard place to look for a package? Can I used PYTHONPATH to coerce cxfreez into looking there for socketpool?

Thanks

Answer 1

I know this thread is old but I just spent 4 days figuring this out and it might be helpful for someone else so here goes:

The description for include_files option in cx_freeze is misleading. If you want to use the include_files option for build_exe here is an example of how you do that

include_files=[
           (r"C:Python27Scriptsmk2ifcoremd.dll",               "mk2ifcoremd.dll"),
           (r"C:Python27Scriptsmk2ifportmd.dll",               "mk2ifportmd.dll"),
           (r"C:Python27Scriptsmk2imalloc.dll",                "mk2imalloc.dll"),
           (r"C:Python27Scriptsmk2iomp5md.dll",                "mk2iomp5md.dll"),
           (r"C:Python27Scriptsmk2mmd.dll",                    "mk2mmd.dll"),]

You must have a list of tuples with an absolute path for the source and ONLY THE FILENAME for the destination. I ran into trouble with cx_freeze because it did/ does not copy all the required DLLs into the dist directory. So my exe tested fine on my machine but did not work on another machine that did not have Python installed. In order to debug this I had to repeatedly move my Python installation out of the way (renamed C:Python27 to C:_Python27) and see if there were missing libraries.

Here is an example setup.py file that works for me:

# invoke using:
#  python setup.py build

from cx_Freeze import setup, Executable

import sys
import glob
import os
import zlib
import shutil

# Remove the existing folders folder
shutil.rmtree("build", ignore_errors=True)
shutil.rmtree("dist", ignore_errors=True)

########################################
# Here is a list of the Executable options
########################################

#"script":               #the name of the file containing the script which is to be frozen
#"initScript":           #the name of the initialization script that will be executed before the actual script is executed; this script is used to set up the environment for the executable; if a name is given without an absolute path the names of files in the initscripts subdirectory of the cx_Freeze package is searched
#"base":                 #the name of the base executable; if a name is given without an absolute path the names of files in the bases subdirectory of the cx_Freeze package is searched
#"path":                 #list of paths to search for modules
#"targetDir":            #the directory in which to place the target executable and any dependent files
#"targetName":           #the name of the target executable; the default value is the name of the script with the extension exchanged with the extension for the base executable
#"includes":             #list of names of modules to include
#"excludes":             #list of names of modules to exclude
#"packages":             #list of names of packages to include, including all of the package's submodules
#"replacePaths":         #Modify filenames attached to code objects, which appear in tracebacks. Pass a list of 2-tuples containing paths to search for and corresponding replacement values. A search for '*' will match the directory containing the entire package, leaving just the relative path to the module.
#"compress":             #boolean value indicating if the module bytecode should be compressed or not
#"copyDependentFiles":   #boolean value indicating if dependent files should be copied to the target directory or not
#"appendScriptToExe":    #boolean value indicating if the script module should be appended to the executable itself
#"appendScriptToLibrary":#boolean value indicating if the script module should be appended to the shared library zipfile
#"icon":                 #name of icon which should be included in the executable itself on Windows or placed in the target directory for other platforms
#"namespacePackages":    #list of packages to be treated as namespace packages (path is extended using pkgutil)
#"shortcutName":         #the name to give a shortcut for the executable when included in an MSI package
#"shortcutDir":          #the directory in which to place the shortcut when being installed by an MSI package; see the MSI Shortcut table documentation for more information on what values can be placed here.
MY_TARGET_EXE = Executable(
    # what to build
    script = "main.py",
    initScript = None,
    base = 'Win32GUI',
    targetDir = r"dist",
    targetName = "MyProgram.exe",
    compress = True,
    copyDependentFiles = True,
    appendScriptToExe = False,
    appendScriptToLibrary = False,
    icon = None
    )

########################################
#Here is a list of the build_exe options
########################################
#1) append the script module to the executable
append_script_to_exe=False
#2) the name of the base executable to use which, if given as a relative path, will be joined with the bases subdirectory of the cx_Freeze installation; the default value is "Console"
base="Console"
#3) list of names of files to exclude when determining dependencies of binary files that would normally be included; note that version numbers that normally follow the shared object extension are stripped prior to performing the comparison
bin_excludes=[]
#4) list of names of files to include when determining dependencies of binary files that would normally be excluded; note that version numbers that normally follow the shared object extension are stripped prior to performing the comparison
bin_includes=[] 
#5) list of paths from which to exclude files when determining dependencies of binary files
bin_path_excludes=[]
#6) list of paths from which to include files when determining dependencies of binary files
bin_path_includes=[]
#7) directory for built executables and dependent files, defaults to build/
build_exe="dist/"
#8) create a compressed zip file
compressed=False
#9) comma separated list of constant values to include in the constants module called BUILD_CONSTANTS in form <name>=<value>
constants=[]
#10) copy all dependent files
copy_dependent_files=True
#11) create a shared zip file called library.zip which will contain all modules shared by all executables which are built 
create_shared_zip=True
#12) comma separated list of names of modules to exclude
excludes = ['_gtkagg', '_tkagg', 'bsddb', 'curses', 'email', 'pywin.debugger',
            'pywin.debugger.dbgcon', 'pywin.dialogs', 'tcl',
            'Tkconstants', 'Tkinter']
#13) include the icon in the frozen executables on the Windows platform and alongside the frozen executable on other platforms
icon=False
#13) comma separated list of names of modules to include
includes = ['sip', 'matplotlib.backends.backend_wxagg']
#15) list containing files to be copied to the target directory; 
#  it is expected that this list will contain strings or 2-tuples for the source and destination; 
#  the source can be a file or a directory (in which case the tree is copied except for .svn and CVS directories); 
#  the target must not be an absolute path
#
# NOTE: INCLUDE FILES MUST BE OF THIS FORM OTHERWISE freezer.py line 128 WILL TRY AND DELETE dist/. AND FAIL!!!
# Here is a list of ALL the DLLs that are included in Python27Scripts 
include_files=[
           (r"C:Python27Scriptsmk2ifcoremd.dll",               "mk2ifcoremd.dll"),
           (r"C:Python27Scriptsmk2ifportmd.dll",               "mk2ifportmd.dll"),
           (r"C:Python27Scriptsmk2imalloc.dll",                "mk2imalloc.dll"),
           (r"C:Python27Scriptsmk2iomp5md.dll",                "mk2iomp5md.dll"),
           (r"C:Python27Scriptsmk2mmd.dll",                    "mk2mmd.dll"),
           (r"C:Python27Scriptsmk2_avx.dll",                   "mk2_avx.dll"),
           (r"C:Python27Scriptsmk2_blacs_ilp64.dll",           "mk2_blacs_ilp64.dll"),
           (r"C:Python27Scriptsmk2_blacs_intelmpi_ilp64.dll",  "mk2_blacs_intelmpi_ilp64.dll"),
           (r"C:Python27Scriptsmk2_blacs_intelmpi_lp64.dll",   "mk2_blacs_intelmpi_lp64.dll"),
           (r"C:Python27Scriptsmk2_blacs_lp64.dll",            "mk2_blacs_lp64.dll"),
           (r"C:Python27Scriptsmk2_blacs_mpich2_ilp64.dll",    "mk2_blacs_mpich2_ilp64.dll"),
           (r"C:Python27Scriptsmk2_blacs_mpich2_lp64.dll",     "mk2_blacs_mpich2_lp64.dll"),
           (r"C:Python27Scriptsmk2_blacs_msmpi_ilp64.dll",     "mk2_blacs_msmpi_ilp64.dll"),
           (r"C:Python27Scriptsmk2_blacs_msmpi_lp64.dll",      "mk2_blacs_msmpi_lp64.dll"),
           (r"C:Python27Scriptsmk2_cdft_core.dll",             "mk2_cdft_core.dll"),
           (r"C:Python27Scriptsmk2_core.dll",                  "mk2_core.dll"),
           (r"C:Python27Scriptsmk2_def.dll",                   "mk2_def.dll"),
           (r"C:Python27Scriptsmk2_intel_thread.dll",          "mk2_intel_thread.dll"),
           (r"C:Python27Scriptsmk2_mc.dll",                    "mk2_mc.dll"),
           (r"C:Python27Scriptsmk2_mc3.dll",                   "mk2_mc3.dll"),
           (r"C:Python27Scriptsmk2_p4n.dll",                   "mk2_p4n.dll"),
           (r"C:Python27Scriptsmk2_pgi_thread.dll",            "mk2_pgi_thread.dll"),
           (r"C:Python27Scriptsmk2_rt.dll",                    "mk2_rt.dll"),
           (r"C:Python27Scriptsmk2_scalapack_ilp64.dll",       "mk2_scalapack_ilp64.dll"),
           (r"C:Python27Scriptsmk2_scalapack_lp64.dll",        "mk2_scalapack_lp64.dll"),
           (r"C:Python27Scriptsmk2_sequential.dll",            "mk2_sequential.dll"),
           (r"C:Python27Scriptsmk2_vml_avx.dll",               "mk2_vml_avx.dll"),
           (r"C:Python27Scriptsmk2_vml_def.dll",               "mk2_vml_def.dll"),
           (r"C:Python27Scriptsmk2_vml_mc.dll",                "mk2_vml_mc.dll"),
           (r"C:Python27Scriptsmk2_vml_mc2.dll",               "mk2_vml_mc2.dll"),
           (r"C:Python27Scriptsmk2_vml_mc3.dll",               "mk2_vml_mc3.dll"),
           (r"C:Python27Scriptsmk2_vml_p4n.dll",               "mk2_vml_p4n.dll"),
# These next DLLs appear to be copied correctly or as needed by cxfreeze...   
#           (r"C:Python27Scriptslibgcc_s_sjlj-1.dll",           "libgcc_s_sjlj-1.dll"),
#           (r"C:Python27Scriptslibgfortran-3.dll",             "libgfortran-3.dll"),
#           (r"C:Python27Scriptslibssp-0.dll",                  "libssp-0.dll"),
#           (r"C:Python27Scriptslibstdc++-6.dll",               "libstdc++-6.dll"),
#           (r"C:Python27Scriptspythoncom27.dll",               "pythoncom27.dll"),
#           (r"C:Python27Scriptspywintypes27.dll",              "pywintypes27.dll"),
            ]
             #,("Microsoft.VC90.MFC", mfcfiles), ]
#16) include the script module in the shared zip file
include_in_shared_zip=True
#17) include the Microsoft Visual C runtime DLLs and (if necessary) the manifest file required to run the executable without needing the redistributable package installed
include_msvcr =False
#18) the name of the script to use during initialization which, if given as a relative path, will be joined with the initscripts subdirectory of the cx_Freeze installation; the default value is "Console"
init_script=""
#19) comma separated list of packages to be treated as namespace packages (path is extended using pkgutil)
namespace_packages=[]
#20) optimization level, one of 0 (disabled), 1 or 2
optimize=0
#21) comma separated list of packages to include, which includes all submodules in the package
packages = ['numpy.linalg']
#22) comma separated list of paths to search; the default value is sys.path
path = []
#23) Modify filenames attached to code objects, which appear in tracebacks. Pass a comma separated list of paths in the form <search>=<replace>. The value * in the search portion will match the directory containing the entire package, leaving just the relative path to the module.
replace_paths=[]
#24) suppress all output except wa