multiprocessing - python multiprocess don't finish properly

Question

My problem is finishing subprocess, I use multiprocess library and in one machine with the return or exit line, the process die before the join, but in another machine not. The processes always grow and neither of them finish after do its job. In both machines the version of python is 2.7.3rc2.

semaphore_processes_limit = BoundedSemaphore(value=PROCS_LIMIT)

# Starting searches
procs = []
for word in words:
    semaphore_processes_limit.acquire()
    p = Process(target=searching, args=(word,))
    procs.append(p)
    p.start()

# Wait for all worker processes to finish
for p in procs:
    p.join()

# Process
def searching(word):
return # or exit(0)

Thank you.

Answer 1

Hi I tried to reproduce the same problem on these two machines:

Python 2.6.7 (r267:88850, Feb 2 2012, 23:50:20) Cygwin on Vista

Python 2.7.3 (default, Aug 1 2012, 05:16:07) Ubuntu 12.04

Both of them finished properly after p.join().

Explanatin 1: However if I lower the PROCS_LIMIT to a number lower then len(words) the last process does not finish.

Explanation 2: The semaphores can be handled differently on different host operating systems. And thus yield diffrent results. See the warning on the top of this page: http://docs.python.org/2/library/multiprocessing.html

Previously I have had problems with the subprocess module in python on Cygwin due to lacking threading support.

Can you describe what machine type and operating system you are running?

This is the modifications I made to your code to make it run:

from multiprocessing import *
from threading import *

# Process
def searching(word):
    print(word)
    return # or exit(0)

PROCS_LIMIT = 5
semaphore_processes_limit = BoundedSemaphore(value=PROCS_LIMIT)

# Starting searches
words = ["foo", "bar", "baz", "buz", "biz"]
procs = []
for word in words:
    semaphore_processes_limit.acquire()
    p = Process(target=searching, args=(word,))
    procs.append(p)
    p.start()

# Wait for all worker processes to finish
for p in procs:
    p.join()