python 3.6 socket pickle data was truncated

Question

i can not send my numpy array in socket, i use pickle but il my client pickle crash with this error: pickle data was truncated

my server : i create numpy array and i want to send in my client with pickle (it's work)

import socket, pickle
import numpy as np
from PIL import ImageGrab
import cv2


while(True):
    HOST = 'localhost'
    PORT = 50007
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4096)
    s.bind((HOST, PORT))
    s.listen(1)
    conn, addr = s.accept()
    print ('Connected by', addr)

    arr = np.array([[0, 1], [2, 3]])
    printscreen_pil=ImageGrab.grab(bbox=(10,10,500,500))
    img = np.array(printscreen_pil) ## Transform to Array

    data_string = pickle.dumps(img)
    conn.send(data_string)

    msg_recu = conn.recv(4096)
    print(msg_recu.decode())

    conn.close()

my client He have my numpy array, but i can not load with pickle, i have this error.

import socket, pickle
import numpy as np

HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))

msg_a_envoyer = "hello".encode()
s.send(msg_a_envoyer)


while 1:
    data = s.recv(4096)
    if not data: break
    data_arr = pickle.loads(data)
    print (data_arr)
s.close()

Answer 1

the problem is that if the size of the pickled data is > 4096 you only get the first part of the pickled data (hence the pickle data was truncated message you're getting)

You have to append the data and pickle it only when the reception is complete, for example like this:

data = b""
while True:
    packet = s.recv(4096)
    if not packet: break
    data += packet

data_arr = pickle.loads(data)
print (data_arr)
s.close()

increasing a bytes object is not very performant, would be better to store the parts in a list of objects, then join, though. Faster variant:

data = []
while True:
    packet = s.recv(4096)
    if not packet: break
    data.append(packet)
data_arr = pickle.loads(b"".join(data))
print (data_arr)
s.close()