python - Filter out rows with more than certain number of NaN

Question

In a Pandas dataframe, I would like to filter out all the rows that have more than 2 NaNs.

Essentially, I have 4 columns and I would like to keep only those rows where at least 2 columns have finite values.

Can somebody advise on how to achieve this?

Answer 1

You have phrased 2 slightly different questions here. In the general case, they have different answers.

I would like to keep only those rows where at least 2 columns have finite values.

df = df.dropna(thresh=2)

This keeps rows with 2 or more non-null values.

---

I would like to filter out all the rows that have more than 2 NaNs

df = df.dropna(thresh=df.shape[1]-2)

This filters out rows with 2 or more null values.

In your example dataframe of 4 columns, these operations are equivalent, since df.shape[1] - 2 == 2. However, you will notice discrepancies with dataframes which do not have exactly 4 columns.

---

Note dropna also has a subset argument should you wish to include only specified columns when applying a threshold. For example:

df = df.dropna(subset=['col1', 'col2', 'col3'], thresh=2)