parallel processing - Using Python multiprocessing to run object computations

Question

I need to do run some computations to update object attributes. I want to use parallel computing since I need to update multiple objects' attributes, that is, I have multiple objects and I need to do the same computation for each one. The objects do not share information between them.

I am currently using a process pool with map or a similar function, and the problem is that these processes copy the object, then do the computation, instead of just doing the computation directly using the original object. Is there any way around this?

As an example:

from multiprocessing import Pool

class A:

    def __init__(self, init):
        self.a = init

    def func(self, b):
        self.a = self.a + b

foo = A(2)
print(foo.a) # prints 2

p = Pool()
result = p.map(foo.func, (3,))
print(foo.a) #prints 2, should print 5

foo.func(3)
print(foo.a) #prints 5 as expected

question from:https://stackoverflow.com/questions/65931142/using-python-multiprocessing-to-run-object-computations

Answer 1

Here's my workaround using queues:

from multiprocessing import Process, Queue
import time

class A:
    def __init__(self, init):
        self.a = init

    def func(self, b):
        self.a = self.a + b

    def par_func(self, q, b):
        print('starting')
        self.func(b)
        q.put(self.a)
        time.sleep(10)
        print('whew! all done')

foo = A(2)
bar = A(6)
print('initial:', foo.a, bar.a) # prints 2, 6

q1, q2 = Queue(), Queue()
p1, p2 = Process(target=foo.par_func, args=(q1, 3)), Process(target=bar.par_func, args=(q2, 4))
p1.start()
p2.start()
p1.join()
p2.join()

#prints from par_func() were nonsequential i.e. parallelism is occurring 

foo.a = q1.get()
bar.a = q2.get()

print('final:', foo.a, bar.a) # prints 5, 10

this is probably far from ideal, but it works for my purposes. I'm leaving it here for anyone else with this problem.