python - Using logical operators in building a Pandas DataFrame

Question

I have two snippets of pandas code which I think should be equivalent, but the second one doesn't do what I expect.

# snippet 1
    data = all_data[[((np.isfinite(all_data[self.design_metric][i]) 
                    and all_data['Source'][i] == 2)) 
                    or ((np.isfinite(all_data[self.actual_metric][i]) 
                    and all_data['Source'][i] != 2))
                    for i in range(len(all_data))]]


# snippet 2
    data = all_data[(all_data['Source'] == 2 &
                    np.isfinite(all_data[self.design_metric])) |
                    (all_data['Source'] != 2 &
                    np.isfinite(all_data[self.actual_metric]))]

Each section (e.g. all_data['Source'] == 2 ) does what I expect on its own but it seems that I'm doing something wrong with the logical operators as the final result is coming out with a different result to the list comprehension version.

Answer 1

The & operator binds more tightly than == (or any comparison operator). See the documentation. A simpler example is:

>>> 2 == 2 & 3 == 3
False

This is because it is grouped as 2 == (2 & 3) == 3, and then comparison chaining is invoked. This is what is happening in your case. You need to put parentheses around each comparison.

 data = all_data[((all_data['Source'] == 2) &
                np.isfinite(all_data[self.design_metric])) |
                ((all_data['Source'] != 2) &
                np.isfinite(all_data[self.actual_metric]))]

Note the extra parentheses around the == and != comparisons.