multiprocessing - Python sharing a dictionary between parallel processes

Question

I want to share a dictionary between my processes as follows:

def f(y,x):
    y[x]=[x*x]                                                          

if __name__ == '__main__':
    pool = Pool(processes=4)
    inputs = range(10)
    y={}                             
    result = pool.map(f,y,inputs)

The y returns {}. How can I make it work?

Thanks,

Answer 1

This looks like you are using the multiprocessing module. You didn't say, and that's an important bit of information.

The .map() function on a multiprocessing.Pool() instance takes two arguments: a function, and a sequence. The function will be called with successive values from the sequence.

You can't collect values in a mutable like a dict (in the example, it's argument y) because your code will be running in multiple different processes. Writing a value to a dict in another process doesn't send that value back to the original process. But if you use Pool.map() the other processes will return the result from each function call, back to the first process. Then you can collect the values to build a dict.

Example code:

import multiprocessing as mp

def f(x):
    return (x, x*x)

if __name__ == '__main__':
    pool = mp.Pool()
    inputs = range(10)
    result = dict(pool.map(f, inputs))

result is set to: {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25, 6: 36, 7: 49, 8: 64, 9: 81}

Let's change it so that instead of computing x*x it will raise x to some power, and the power will be provided. And let's make it take a string key argument. This means that f() needs to take a tuple argument, where the tuple will be (key, x, p) and it will compute x**p.

import multiprocessing as mp

def f(tup):
    key, x, p = tup  # unpack tuple into variables
    return (key, x**p)

if __name__ == '__main__':
    pool = mp.Pool()
    inputs = range(10)
    inputs = [("1**1", 1, 1), ("2**2", 2, 2), ("2**3", 2, 3), ("3**3", 3, 3)]
    result = dict(pool.map(f, inputs))

If you have several sequences and you need to join them together to make a single sequence for the above, look into using zip() or perhaps itertools.product.