python - merge rows pandas dataframe based on condition

Question

Hi have a dataframe df

containing a set of events (rows).

df = pd.DataFrame(data=[[1, 2,   7, 10],
                   [10, 22, 1, 30],
                   [30, 42, 2, 10],  
                   [100,142, 22,1],
                   [143, 152, 2, 10],
                   [160, 162, 12, 11]],columns=['Start','End','Value1','Value2'])

 df
Out[15]: 
   Start  End  Value1  Value2
0      1    2       7      10
1     10   22       1      30
2     30   42       2      10
3    100  142      22       1
4    143  152       2      10
5    160  162      12      11

If 2 (or more) consecutive events are <= 10 far apart I would like to merge the 2 (or more) events (i.e. use the start of the first event, end of the last and sum the values in Value1 and Value2).

In the example above df becomes:

 df
Out[15]: 
   Start  End  Value1  Value2
0      1   42      10      50
1    100  162      36      22

Answer 1

That's totally possible:

df.groupby(((df.Start  - df.End.shift(1)) > 10).cumsum()).agg({'Start':min, 'End':max, 'Value1':sum, 'Value2': sum})

Explanation:

start_end_differences = df.Start  - df.End.shift(1) #shift moves the series down
threshold_selector = start_end_differences > 10 # will give you a boolean array where true indicates a point where the difference more than 10.
groups = threshold_selector.cumsum() # sums up the trues (1) and will create an integer series starting from 0
df.groupby(groups).agg({'Start':min}) # the aggregation is self explaining

---

Here is a generalized solution that remains agnostic of the other columns:

cols = df.columns.difference(['Start', 'End'])
grps = df.Start.sub(df.End.shift()).gt(10).cumsum()
gpby = df.groupby(grps)
gpby.agg(dict(Start='min', End='max')).join(gpby[cols].sum())

   Start  End  Value1  Value2
0      1   42      10      50
1    100  162      36      22