python - multiprocessing in IPython console on Windows machine - if __name_ requirement

Question

I'm working with IPython and Spyder IDE on a Windows machine. When the IDE is starting, a set of py-files is loaded to define some functions that make my work a bit easier. Everything works as expected.

Now I would like to upgrade one of these function to use multiprocessing, but on Windows this requires the if __name__ == "__main__": statement. So it seems that I cannot call the function directly and pass the arguments from the IPython console.

For example one of the py-files (let's call it test.py) could look like the following code.

import multiprocessing as mp
import random
import string

# define a example function
def rand_string(length, output):
    """ Generates a random string of numbers, lower- and uppercase chars. """
    rand_str = ''.join(random.choice(
                string.ascii_lowercase
                + string.ascii_uppercase
                + string.digits)
           for i in range(length))
    output.put(rand_str)


def myFunction():
    # Define an output queue
    output = mp.Queue()        

    # Setup a list of processes that we want to run
    processes = [mp.Process(target=rand_string, args=(5, output)) for x in range(4)]

    # Run processes
    for p in processes:
        p.start()

    # Exit the completed processes
    for p in processes:
        p.join()

    # Get process results from the output queue
    results = [output.get() for p in processes]

    print(results)

In my IPython console I would like to use the line

myFunction()

to trigger all the calculations. But on Windows a end up getting a BrokenPipe error.

When I put

if __name__ == "__main__":
     myFunction()

at the end of the py-file and run the complete file by

runfile(test.py)

it works. Of course. But that makes it very hard to pass arguments to the function as I always have to edit the test.py-file itself.

My question is: How do I get the multiprocessing function running without putting it in this if __name__ == "__main__": statement??

Answer 1

So, I solved that specific problem.

1. Put the defintion of rand_string in a separate file, called test2.

2. Import test2 as module into my test.py script

   import test2 as test2

3. modify the following line to access the test2 module

   processes = [mp.Process(target=test2.rand_string, args=(5, output)) for x in range(4)]
   

4. Run test.py

5. Call myFunction()

6. Be Happy :)

The solution is based on this multiprocessing tutorial that suggests to import the target function from another script. This solution bypasses the safe self import by the if __name__ -wrapper to get access to the target function.