python - Repeatedly write to STDIN and read STDOUT of a Subprocess without closing it

Question

I am trying to employ a Subprocess in Python for keeping an external script open in a Server-like fashion. The external script first loads a model. Once this is done, it accepts requests via STDIN and returns processed strings to STDOUT.

So far, I've tried

tokenizer = subprocess.Popen([tokenizer_path, '-l', lang_prefix], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

However, I cannot use

tokenizer.stdin.write(input_string+'
')
out = self._tokenizer.stdout.readline()

in order to repeatedly process input_strings by means of the subprocess –?out will just be empty, no matter if I use stdout.read() or stdout.readline(). However, it works when I close the stdin with tokenizer.stdin.close() before reading STDOUT, but this closes the subprocess, which is not what I want as I would have to reload the whole external script again before sending another request.

Is there any way to use a subprocess in a server-like fashion in python without closing and re-opening it?

Answer 1

Thanks to this Answer, I found out that a slave handle must be used in order to properly communicate with the subprocess:

master, slave = pty.openpty()
tokenizer = subprocess.Popen(script, shell=True stdin=subprocess.PIPE, stdout=slave)
stdin_handle = process.stdin
stdout_handle = os.fdopen(master)

Now, I can communicate to the subprocess without closing it via

stdin_handle.write(input)
stdout_handle.readline() #gets the processed input