piping binary data between python and c++

Question

I have been working on a plugin to QGIS using Python 2.7 which works fine until I actually go to do some image processing on the mapped layers. Even simple tasks like collecting the RGB values of the raster layer (say a 5k x 1k section) take a bit (~2 minutes). I could live with that if I had to, but when I start calculating simple metrics on the data (like Entropy) the amount of time needed for processing explodes (I stopped the processing after ~40 unresponsive minutes).

I've written the Entropy code before in C++ and would love to be able to just process the data that way. After doing some research I found that Python can use stdin and stdout to pipe data and came across the following example buried in a forum at http://ubuntuforums.org/archive/index.php/t-524072.html

using this Python code as a driver

import subprocess

proc = subprocess.Popen("C:Python27PythonPipes.exe",
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

state = "run"
while state == "run":
    input = raw_input("Message to CPP>> ")

    if input == "quit":
        state = "terminate" # any string other than "run" will do

    proc.stdin.write(input + "
")
    cppMessage = proc.stdout.readline().rstrip("
") 
    print "cppreturn message ->" + cppMessage + " written by python 
"

and this c++ code as a data processer

#include <iostream>
#include <string>

using namespace std;

int main(int argc, char* args[]){

    string python_message = "";
    bool quit = false;

    while (!quit){
        cin >> python_message;

        if (python_message == "quit"){
            quit = true;
        }
        else if (python_message == "first"){
            cout << "First Hello!" << endl;
        }
        else if (python_message == "second"){
            cout << "Second Hello!" << endl;
        }
        else if (python_message == "third"){
            cout << "Third Hello!" << endl;
        }
        else {
            cout << "Huh?" << endl;
        }
    }
    return 0;
}

This code works great for text data. I can pipe text back and forth all day long with it. but what I want to do is pass binary data back and forth so that I can send the integer raster layer data from python to c++ for processing, and then return the results.

I have looked around and tried various combinations of putting a buffer of bytes into

proc.stdin.write(bufferdata)

and using

fread(Arraypointer, 4,1,stdin) 

for receiving the buffer data on the c++ program side as well as

fwrite(outArraypointer,4,1,stdout)

to pipe data back to python, but have nopt been successful. I think that part of the problem may be that the text version uses cin and waits for the EOL indicator. I am unclear how to do something similar in the case of Binary data.

I would like to know how to modify the example code above to send an int from a python program to a c++ program. increment the int in the c++ program, and then send that int back to the python program. please bear in mind that I am going to have to do this with millions of ints at a time in case that caveat ways on your proposed solution.

If this doesn't work out I'll move to just having python write out a binary file that the c++ code reads in, but I'd really like to use this approach if it's possible.

Thanks for the help ahead of time.

Update Roland's solution was the starting point I needed. For those who come later, a working prototype of the code I described above is below. It is a slight modification of Roland's

Python Driver:

import subprocess

proc = subprocess.Popen("C:Python27PythonPipes.exe",
stdin=subprocess.PIPE,stdout=subprocess.PIPE)

s1 = bytearray(10)   

s1[0] = 65 #A
s1[1] = 66 #B
s1[2] = 67 #C
s1[3] = 68 #D
s1[4] = 69 #E
s1[5] = 70 #F
s1[6] = 71 #G
s1[7] = 72 #H
s1[8] = 73 #I

t = buffer(s1)       

proc.stdin.write(t)
value = [0,0,0,0,0,0,0,0]
for i in range(8):
    value[i] = ord(proc.stdout.read(1))
    print "value i -> " + str(value[i])

proc.stdin.write('q')
proc.wait()

C++ Processor

#include <stdio.h>
char increase;
int main(int argc, char **argv) {
    for (;;) {
        char buf;
        fread(&buf, 1, 1, stdin);
        if ('q' == buf)
            break;
        increase = buf + 1;
        fwrite(&increase, 1, 1, stdout);
        fflush(stdout);
    }

    return 0;
}

Answer 1

The problem probably is buffering: by default, the C stdio buffers everything written to stdout and flushes the buffer back into the pipe only when a newline is written (line buffering). The problem disappears when you call fflush(stdout) after writing. You can also disable (or control) buffering via the setvbuf function defined in <stdio.h>, e.g. use setvbuf(stdout, NULL, _IONBF, 0) to disable buffering completely.

I have tested the first variant with the following two programs:

import subprocess

proc = subprocess.Popen("./echotest",
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE)

proc.stdin.write('abc')
message = proc.stdout.read(3)
print "return message ->" + message + " written by python 
" 
proc.stdin.write('q')
proc.wait()

and a little C program:

#include <stdio.h>

int main (int argc, char **argv) {
    for (;;) {
        char buf;
        fread(&buf, 1, 1, stdin);
        if ('q' == buf)
            break;
        fwrite(&buf, 1, 1, stdout);
        fflush(stdout);
    }

    return 0;
}

Note that you have to specify how many bytes you want to read back from the subprocess or the program will block, waiting for more output. If this bothers you, try one of the solutions to this question.