parallel processing - Deliberately make an orphan process in python

Question

I have a python script (unix-like, based on RHEL), called MyScript, that has two functions, called A and B. I'd like them to run in different, independent processes (detach B and A):

• Start script MyScript
• Execute function A
• Spawn a new process, passing data from function A to B
• While function B runs, continue with function A
• When function A completes, exit MyScript even if B is still running

I thought I should use multiprocessing to create a daemon process, but the documentation suggests that's not the right usecase. So, I decided to spawn a child process and child^2 process (the child's child), and then force the child to terminate. While this workaround appears to work, it seems really ugly.

Can you help me make it more pythonic? Does the subprocess module have a method that will operate on a function? Sample code below.

import multiprocessing
import time
import sys
import os

def parent_child():
    p = multiprocessing.current_process()
    print 'Starting parent child:', p.name, p.pid
    sys.stdout.flush()
    cc = multiprocessing.Process(name='childchild', target=child_child)
    cc.daemon = False
    cc.start()
    print 'Exiting parent child:', p.name, p.pid
    sys.stdout.flush()

def child_child():
    p = multiprocessing.current_process()
    print 'Starting child child:', p.name, p.pid
    sys.stdout.flush()
    time.sleep(30)
    print 'Exiting child child:', p.name, p.pid
    sys.stdout.flush()

def main():
    print 'starting main', os.getpid()
    d = multiprocessing.Process(name='parentchild', target=parent_child)
    d.daemon = False
    d.start()
    time.sleep(5)
    d.terminate()
    print 'exiting main', os.getpid()

main()

Answer 1

Here is just a random version of your original code that moves the functionality into a single call spawn_detached(callable). It keeps the detached process running even after the program exits:

import time
import os
from multiprocessing import Process, current_process


def spawn_detached(callable):
    p = _spawn_detached(0, callable)
    # give the process a moment to set up
    # and then kill the first child to detach
    # the second.
    time.sleep(.001)
    p.terminate()

def _spawn_detached(count, callable):
    count += 1
    p = current_process()
    print 'Process #%d: %s (%d)' % (count, p.name, p.pid)

    if count < 2:
        name = 'child'
    elif count == 2:
        name = callable.func_name
    else:
        # we should now be inside of our detached process
        # so just call the function
        return callable()

    # otherwise, spawn another process, passing the counter as well
    p = Process(name=name, target=_spawn_detached, args=(count, callable)) 
    p.daemon = False 
    p.start()
    return p

def operation():
    """ Just some arbitrary function """
    print "Entered detached process"
    time.sleep(15)
    print "Exiting detached process"


if __name__ == "__main__":
    print 'starting main', os.getpid()
    p = spawn_detached(operation)
    print 'exiting main', os.getpid()