python - Why does my pandas filters work in separate steps but not in one command?

Question

I'm having problems filtering my pandas dataframe in one command. For instance, the following multi-step filter works perfectly:

check2020 = check[check['effyear'] == '2020']
check2020_can = check2020[check2020['canceled'] == 'Y']
check2020_can_6 = check2020_can[check2020_can['decile'] == 6]
check2020_can_6_p_s = check2020_can_6[(check2020_can_6['usage'] == 'P') | (check2020_can_6['usage'] == 'S')]

But if I were to do the following:

check[(check['effyear'] == '2020') & (check['canceled'] == 'Y') & (check['decile'] == 6) & (check['usage'] == 'P') | (check['usage'] == 'S')]

Then I get 30,000 more observations that do not all follow the filter. Is the all in one filter not structured correclty?

question from:https://stackoverflow.com/questions/65941981/why-does-my-pandas-filters-work-in-separate-steps-but-not-in-one-command

Answer 1

I believe you need extra pair of parenthesis:

check[(check['effyear'] == '2020') & 
      (check['canceled'] == 'Y') & 
      (check['decile'] == 6) & 
      ((check['usage'] == 'P') | (check['usage'] == 'S')) # () here
     ]

Also, use isin for better syntax:

check[(check['effyear'] == '2020') & 
      (check['canceled'] == 'Y') & 
      (check['decile'] == 6) & 
      check['usage'].isin(['P', 'S'])  # this
     ]