python - process socket data that ends with a line break

Question

What is the best approach to process a socket connection where I need var data to end with a line break ? I'm using the code below but sometimes the tcp packets get chunked and it takes a long time to match data.endswith(" "). I've also tried other approaches, like saving the last line if it doesn't end with and append it to dataon the next loop. but this also doesn't work because multiple packets get chunked and the 1st and 2nd part don't match. I've no control over the other end, it basically sends multiple lines that end in .

Any suggestion will be welcome, as I don't have much knowledge on socket connections.

def receive_bar_updates(s):
    global all_bars
    data = ''
    buffer_size = 4096
    while True:
        data += s.recv(buffer_size)
        if not data.endswith("
"):
            continue
        lines = data.split("
")
        lines = filter(None, lines)
        for line in lines:
            if line.startswith("BH") or line.startswith("BC"):
                symbol = str(line.split(",")[1])
                all_bars[symbol].append(line)
                y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
                y.start()
        data = ""

Example of "normal" data:

line1

line2

line3

Example of chunked data:

line1

line2

lin

Answer 1

If you have a raw input that you want to process as line, the io module is your friend because it will do the low level assembling of packets in lines.

You could use:

class SocketIO(io.RawIOBase):
    def __init__(self, sock):
        self.sock = sock
    def read(self, sz=-1):
        if (sz == -1): sz=0x7FFFFFFF
        return self.sock.recv(sz)
    def seekable(self):
        return False

It is more robust than endswith(' ') because if one packet contains an embedded newline ('ab cd'), the io module will correctly process it. Your code could become:

def receive_bar_updates(s):
    global all_bars
    data = ''
    buffer_size = 4096
    fd = SocketIO(s)  # fd can be used as an input file object

    for line in fd:
        if should_be_rejected_by_filter(line): continue # do not know what filter does...
        if line.startswith("BH") or line.startswith("BC"):
            symbol = str(line.split(",")[1])
            all_bars[symbol].append(line)
            y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
            y.start()