list - Memory leakage issue in python tuples

Question

The identities list contains a big array of approximately 57000 images. Now, I am creating a negative list with the help of itertools.product. This whole list store in RAM which is very costly and my system will be hanged after 4 minutes. How I can optimize the below code and avoid saving in RAM?

for i in range(0, len(idendities) - 1):
    for j in range(i + 1, len(idendities)):
        cross_product = itertools.product(samples_list[i], samples_list[j])
        cross_product = list(cross_product)

        for cross_sample in cross_product:
            negative = []
            negative.append(cross_sample[0])
            negative.append(cross_sample[1])
            negatives.append(negative)
            print(len(negatives))

negatives = pd.DataFrame(negatives, columns=["file_x", "file_y"])
negatives["decision"] = "No"

negatives = negatives.sample(positives.shape[0])

Answer 1

The product from itertools is a generator so naturally it dose not store the whole list in memory, but in the next line, cross_product = list(cross_product) you convert it to list object which store the whole data in your memory.

The idea of a generator is that you don't do all the calculation at the same time, as you do with your call list(itertools.product(samples_list[i], samples_list[j])). So what you want to do is generate the results one by one:

Try something like this:

for i in range(len(idendities) - 1):
    for j in range(i + 1, len(idendities)):
        for cross_sample in itertools.product(samples_list[i], samples_list[j]):
            # do something ...

Some credits to @9mat, @cybot and this question How to get Cartesian product in Python using a generator?