multiprocessing - Python join a process without blocking parent

Question

I'm writing a program that will watch a particular directory for new files containing download URLs. Once a new file is detected, it will create a new process to do the actual download while the parent continues to watch the directory. I'm using the Process interface from multiprocessing. The problem I have is that unless I call process.join() the child process is still running, but process.join() is a blocking function which defeats the purpose of creating the child to handle the actual download.

My question is, is there a way to join the child process in a non-blocking manner which will allow the parent to keep doing its thing?

Partial code:

def main(argv):
  # parse command line args
  ...
  # set up variables
  ...
  watch_dir(watch_dir, download_dir)


def watch_dir(wDir, dDir):
  # Grab the current watch directory listing
  before = dict([(f, None) for f in os.listdir (wDir)])

  # Loop FOREVER
  while 1:
    # sleep for 10 secs
    time.sleep(10)

    # Grab the current dir listing
    after = dict([(f, None) for f in os.listdir (wDir)])

    # Get the list of new files
    added = [f for f in after if not f in before]
    # Get the list of deleted files
    removed = [f for f in before if not f in after]

    if added:
      # We have new files, do your stuff
      print "Added: ", ", ".join(added)

      # Call the new process for downloading
      p = Process(target=child, args=(added, wDir, dDir))
      p.start()
      p.join()

    if removed:
      # tell the user the file was deleted
      print "Removed: ", ", ".join(removed)

    # Set before to the current
    before = after

def child(filename, wDir, dDir):
  # Open filename and extract the url
  ...
  # Download the file and to the dDir directory
  ...
  # Delete filename from the watch directory
  ...
  # exit cleanly
  os._exit(0)

The parent waits for the child to finish execution before continuing after p.join() which is (as far as I can tell) correct. But that defeats the whole purpose of creating the child. If I leave off p.join() then the child remains active and a ps ax | grep python give me 'python <defunct>'.

I'd like the child to finish up what its doing and go away without holding up the parent. Is there a way to do it?

Answer 1

You can set up a separate thread which does the joining. Have it listen on a queue into which you push the subprocess handles:

class Joiner(Thread):
    def __init__(self, q):
        self.__q = q
    def run(self):
        while True:
            child = self.__q.get()
            if child == None:
                return
            child.join()

Then, instead of p.join(), do joinq.put(p) and do a joinq.put(None) to signal the thread to stop. Make sure you use a FIFO queue.