filestream - Python read stream

Question

I need a very inexpensive way of reading a buffer with no terminating string (a stream) in Python. This is what I have, but it wastes a a lot of CPU time and effort. Because it is constantly "trying and catching." I really need a new approach.

Here is a reduced working version of my code:

#! /usr/bin/env/ python
import fcntl, os, sys

if __name__ == "__main__":
    f = open("/dev/urandom", "r")
    fd = f.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

    ready = False
    line = ""
    while True:
        try:
            char = f.read()
            if char == '
':
                continue
            elif char = '
':
                ready = True
            else:
                line += char
        except:
            continue
        if ready:
            print line

Don't run this in the terminal. It's simply for illustration. "urandom" will break your terminal because it spits out a lot of random characters that the terminal emulator interprets no matter what (which can change your current shells settings, title, etc). I was reading from a gps connected via usb.

The problem: this uses 100% of the CPU usage when it can. I have tried this:

#! /usr/bin/env/ python
import fcntl, os, sys

if __name__ == "__main__":
    f = open("/dev/urandom", "r")
    fd = f.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

    for line in f.readlines():
        print line

However, I get IOError: [Errno 11] Resource temporarily unavailable. I have tried to use Popen amongst other things. I am at a loss. Can someone please provide a solution (and please explain everything, as I am not a pro, per se). Also, I should note that this is for Unix (particularly Linux, but it must be portable across all versions of Linux).

Answer 1

You will want to set your buffering mode to the size of the chunk you want to read when opening the file stream. From python documentation:

io.open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True)

"buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size of a fixed-size chunk buffer."

You also want to use the readable() method in the while loop, to avoid unnecessary resource consumption.

However, I advise you to use buffered streams such as io.BytesIO or io.BufferedReader

More info in the docs.