python - Pass multiple parameters to concurrent.futures.Executor.map?

Question

The concurrent.futures.Executor.map takes a variable number of iterables from which the function given is called. How should I call it if I have a generator that produces tuples that are normally unpacked in place?

The following doesn't work because each of the generated tuples is given as a different argument to map:

args = ((a, b) for (a, b) in c)
for result in executor.map(f, *args):
    pass

Without the generator, the desired arguments to map might look like this:

executor.map(
    f,
    (i[0] for i in args),
    (i[1] for i in args),
    ...,
    (i[N] for i in args),
)

Answer 1

One argument that is repeated, one argument in c

from itertools import repeat
for result in executor.map(f, repeat(a), c):
    pass

Need to unpack items of c, and can unpack c

from itertools import izip
for result in executor.map(f, *izip(*c)):
    pass

Need to unpack items of c, can't unpack c

1. Change f to take a single argument and unpack the argument in the function.

2. If each item in c has a variable number of members, or you're calling f only a few times:

   executor.map(lambda args, f=f: f(*args), c)
   

   It defines a new function that unpacks each item from c and calls f. Using a default argument for f in the lambda makes f local inside the lambda and so reduces lookup time.

3. If you've got a fixed number of arguments, and you need to call f a lot of times:

   from collections import deque
   def itemtee(iterable, n=2):
       def gen(it = iter(iterable), items = deque(), next = next):
           popleft = items.popleft
           extend = items.extend
           while True:
               if not items:
                   extend(next(it))
               yield popleft()
       return [gen()] * n
   
   executor.map(f, *itemtee(c, n))
   

Where n is the number of arguments to f. This is adapted from itertools.tee.