python - subprocess.Popen taking too long on WSL Linux

Question

I have this subprocess.Popen() context manager:

with Popen(
    args=command, shell=False, stdout=PIPE, bufsize=1, universal_newlines=True
) as process:

    # TIMING
    start = timer()
    lines = list(process.stdout)
    end = timer()
    print('Time taken:', end - start) # 53.662078000000065 seconds -> Linux

    for _ in tqdm(iterable=lines, total=len(lines)):
        sleep(0.1)

if process.returncode != 0:
    raise CalledProcessError(returncode=process.returncode, cmd=process.args)

And it seems to take 53 seconds to process list(process.stdout) when running in a WSL Linux enviorment. However, when I run it in a Windows enviorment, it only takes 0.6 seconds. I'm finding it strange to see why the timings are so different.

I've tried using subprocess.run() and subprocess.check_output() instead, but they still lead to the same long lag before processing the tqdm() loop.

Am I missing something here? I've tried looking at the docs to see what are the differences using subprocess.Popen() in a Windows vs WSL Linux enviorment, but I'm still unsure what the issue is. Perhaps list(process.stdout) is unnecessary here, and there is a better way to store the lines from stdout.

Any sort of guidance would be very helpful here.

Answer 1

You will need to reevaluate that performance issue in Q3 2019, with WSL2.

See "Announcing WSL 2" from Craig Loewen

Changes in this new architecture will allow for: dramatic file system performance increases, and full system call compatibility, meaning you can run more Linux apps in WSL 2 such as Docker.

File intensive operations like git clone, npm install, apt update, apt upgrade, and more will all be noticeably faster.
The actual speed increase will depend on which app you’re running and how it is interacting with the file system.
Initial tests that we’ve run have WSL 2 running up to 20x faster compared to WSL 1 when unpacking a zipped tarball, and around 2-5x faster when using git clone, npm install and cmake on various projects.

Linux binaries use system calls to perform many functions such as accessing files, requesting memory, creating processes, and more.
In WSL 1 we created a translation layer that interprets many of these system calls and allows them to work on the Windows NT kernel. However, it’s challenging to implement all of these system calls, resulting in some apps being unable to run in WSL 1.
Now that WSL 2 includes its own Linux kernel it has full system call compatibility.