python - Pandas iterating over multiple rows at once with overlap

Question

I have a pandas DataFrame that need to be fed in chunks of n-rows into downstream functions (print in the example). The chunks may have overlapping rows.

Let's start from a dummy DataFrame:

d = {'A':list(range(1000)), 'B':list(range(1000))}
df=pd.DataFrame(d)

In the case of a 2-rows chunks with 1-row overlap I have the following code:

a = df.index.values[:-1]
for i in a:
    print(df.iloc[i:i+2])

The output is something like this:

...
       A    B
996  996  996
997  997  997
       A    B
997  997  997
998  998  998
       A    B
998  998  998
999  999  999

Which is exactly what I want.

Is there a better/faster approach to iterate over chunks of n-rows of a pandas.DataFrame?

Answer 1

Use DataFrame.groupby with integer division with helper 1d array created with same length like df - index values are not overlapped:

d = {'A':list(range(5)), 'B':list(range(5))}
df=pd.DataFrame(d)

print (np.arange(len(df)) // 2)
[0 0 1 1 2]

for i, g in df.groupby(np.arange(len(df)) // 2):
    print (g)

   A  B
0  0  0
1  1  1
   A  B
2  2  2
3  3  3
   A  B
4  4  4

EDIT:

For overlapping values is edited this answer:

def chunker1(seq, size):
    return (seq.iloc[pos:pos + size] for pos in range(0, len(seq)-1))

for i in chunker1(df,2):
    print (i)

   A  B
0  0  0
1  1  1
   A  B
1  1  1
2  2  2
   A  B
2  2  2
3  3  3
   A  B
3  3  3
4  4  4