multiprocessing - Fastest way to extract tar files using Python

Question

I have to extract hundreds of tar.bz files each with size of 5GB. So tried the following code:

import tarfile
from multiprocessing import Pool

files = glob.glob('D:\*.tar.bz') ##All my files are in D
for f in files:

   tar = tarfile.open (f, 'r:bz2')
   pool = Pool(processes=5)

   pool.map(tar.extractall('E:\') ###I want to extract them in E
   tar.close()

But the code has type error: TypeError: map() takes at least 3 arguments (2 given)

How can I solve it? Any further ideas to accelerate extracting?

Answer 1

Define a function that extract a single tar file. Pass that function and a tar file list to multiprocessing.Pool.map:

from functools import partial
import glob
from multiprocessing import Pool
import tarfile


def extract(path, dest):
    with tarfile.open(path, 'r:bz2') as tar:
        tar.extractall(dest)

if __name__ == '__main__':
    files = glob.glob('D:\*.tar.bz')
    pool = Pool(processes=5)
    pool.map(partial(extract, dest='E:\'), files)