python - Howto: workaround of close_fds=True and redirect stdout/stderr on windows

Question

I had a problem: using Python 2.7, it was not possible to create a subprocess using

subprocess.Popen([.......], close_fds=True, stdout=subprocess.PIPE, ...)

On windows, because of limitations. The use of close_fds was needed in my case, as I did not wanted the subprocess to inherit from already opened files file-descriptor. This was called in a library, which meant I had no control over already opened- file descriptors (N flag).

This is a known bug, which was fixed on Python 3.4+

The question I had was: How do I use subprocess without getting

close_fds is not supported on Windows platforms if you redirect stdin/stdout/stderr

Answer below

Answer 1

This issue is fixed on Python 3.7+ by default

This is definatly a tricky hack: answer is to iterate through already opened file descriptors, just before using the subprocess module.

def _hack_windows_subprocess():
    """HACK: python 2.7 file descriptors.
    This magic hack fixes https://bugs.python.org/issue19575
    by adding HANDLE_FLAG_INHERIT to all already opened file descriptors.
    """
    # See https://github.com/secdev/scapy/issues/1136
    import stat
    from ctypes import windll, wintypes
    from msvcrt import get_osfhandle

    HANDLE_FLAG_INHERIT = 0x00000001

    for fd in range(100):
        try:
            s = os.fstat(fd)
        except:
            continue
        if stat.S_ISREG(s.st_mode):
            handle = wintypes.HANDLE(get_osfhandle(fd))
            mask   = wintypes.DWORD(HANDLE_FLAG_INHERIT)
            flags  = wintypes.DWORD(0)
            windll.kernel32.SetHandleInformation(handle, mask, flags)

Here is a sample that would crash without it:

import os, subprocess
f = open("a.txt", "w")
subprocess.Popen(["cmd"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
f.close()
os.remove(f.name)

Traceback (most recent call last):

File "stdin", line 1, in module

WindowsError: [Error 32] Le processus ne peut pas accúder au fichier car ce fichier est utilisú par un autre processus: 'a.txt'

Now with the fix:

import os, subprocess
f = open("a.txt", "w")
_hack_windows_subprocess()
subprocess.Popen(["cmd"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
f.close()
os.remove(f.name)

Works.

Hope i helped