python - PyInstaller-built Windows EXE fails with multiprocessing

Question

In my project I'm using Python's multiprocessing library to create multiple processes in __main__. The project is being packaged into a single Windows EXE using PyInstaller 2.1.1.

I create new processes like so:

from multiprocessing import Process
from Queue import Empty

def _start():
    while True:
        try:
            command = queue.get_nowait()
        # ... and some more code to actually interpret commands
        except Empty:
            time.sleep(0.015)

def start():
    process = Process(target=_start, args=args)
    process.start()
    return process

And in __main__:

if __name__ == '__main__':
    freeze_support()

    start()

Unfortunately, when packaging the application into an EXE and launching it, I get WindowsError 5 or 6 (seems random) at this line:

command = queue.get_nowait()

A recipe at PyInstaller's homepage claims that I have to modify my code to enable multiprocessing in Windows when packaging the application as a single file.

I'm reproducing the code here:

import multiprocessing.forking
import os
import sys


class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            # Last character is stripped in C-loader. We have to add
            # '/' or '\' at the end.
            os.putenv('_MEIPASS2', sys._MEIPASS + os.sep)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')


class Process(multiprocessing.Process):
    _Popen = _Popen


class SendeventProcess(Process):
    def __init__(self, resultQueue):
        self.resultQueue = resultQueue

        multiprocessing.Process.__init__(self)
        self.start()

    def run(self):
        print 'SendeventProcess'
        self.resultQueue.put((1, 2))
        print 'SendeventProcess'


if __name__ == '__main__':
    # On Windows calling this function is necessary.
    if sys.platform.startswith('win'):
        multiprocessing.freeze_support()
    print 'main'
    resultQueue = multiprocessing.Queue()
    SendeventProcess(resultQueue)
    print 'main'

My frustration with this "solution" is that, one, it's absolutely unclear what exactly it is patching, and, two, that it's written in such a convoluted way that it becomes impossible to infer which parts are the solution, and which are just an illustration.

Can anyone share some light on this issue, and provide insight what exactly needs to be changed in a project that enables multiprocessing in PyInstaller-built single-file Windows executables?

Answer 1

To add on to nikola's answer...

*nix (Linux, Mac OS X, etc.) does NOT require any changes for PyInstaller to work. (This includes both --onedir and --onefile options.) If you only intend to support *nix systems, no need to worry about any of this.

However, if you are planning on supporting Windows, you will need to add some code, depending on which option you pick: --onedir or --onefile.

If you plan to use --onedir, all you will need to add is a special method call:

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

According to the documentation, this call must be made immediately after if __name__ == '__main__':, or else it will not work. (It is strongly suggested that you have these two lines in your main module.)

In reality, however, you can afford to do a check before the call, and things will still work:

if __name__ == '__main__':
    if sys.platform.startswith('win'):
        # On Windows calling this function is necessary.
        multiprocessing.freeze_support()

However, calling multiprocessing.freeze_support() is possible on other platforms and situations as well - running it only impacts freezing support on Windows. If you're a bytecode nut, you'll notice that the if statement adds some bytecode, and makes potential savings from using an if statement negligible. Therefore, you should just stick to a simple multiprocessing.freeze_support() call immediately after if __name__ == '__main__':.

If you plan to use --onefile, you will need to add nikola's code:

import multiprocessing.forking
import os
import sys

class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            os.putenv('_MEIPASS2', sys._MEIPASS)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')

class Process(multiprocessing.Process):
    _Popen = _Popen

# ...

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    # Use your new Process class instead of multiprocessing.Process

You can combine the above with the rest of his code, or the following:

class SendeventProcess(Process):
    def __init__(self, resultQueue):
        self.resultQueue = resultQueue

        multiprocessing.Process.__init__(self)
        self.start()

    def run(self):
        print 'SendeventProcess'
        self.resultQueue.put((1, 2))
        print 'SendeventProcess'

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    print 'main'
    resultQueue = multiprocessing.Queue()
    SendeventProcess(resultQueue)
    print 'main'

I got the code from here, PyInstaller's new site for the multiprocessing recipe. (They seem to have shut down their Trac based site.)

Note that they have a minor error with their code for --onefile multiprocessing support. They add os.sep to their _MEIPASS2 environment variable. (Line: os.putenv('_MEIPASS2', sys._MEIPASS + os.sep)) This breaks things:

  File "<string>", line 1
    sys.path.append(r"C:UsersAlbertAppDataLocalTemp\_MEI14122")
                                                                    ^
SyntaxError: EOL while scanning string literal

The code I provided above is the same, without the os.sep. Removing the os.sep fixes this issue and allows multiprocessing to work using the --onefile configuration.

In summary:

Enabling --onedir multiprocessing support on Windows (does NOT work with --onefile on Windows, but otherwise safe on all platforms/configurations):

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

Enabling --onefile multiprocessing support on Windows (safe on all platforms/configurations, compatible with --onedir):

import multiprocessing.forking
import os
import sys

class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            os.putenv('_MEIPASS2', sys._MEIPASS)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')

class Process(multiprocessing.Process):
    _Popen = _Popen

# ...

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    # Use your new Process class instead of multiprocessing.Process

Sources: PyInstaller Recipe, Python multiprocessing docs