python - How to conditionally remove duplicates from a pandas dataframe

Question

Consider the following dataframe

import pandas as pd
df = pd.DataFrame({'A' : [1, 2, 3, 3, 4, 4, 5, 6, 7],
                   'B' : ['a','b','c','c','d','d','e','f','g'],
                   'Col_1' :[np.NaN, 'A','A', np.NaN, 'B', np.NaN, 'B', np.NaN, np.NaN],
                   'Col_2' :[2,2,3,3,3,3,4,4,5]})
df
Out[92]: 
    A  B Col_1  Col_2
 0  1  a   NaN      2
 1  2  b     A      2
 2  3  c     A      3
 3  3  c   NaN      3
 4  4  d     B      3
 5  4  d   NaN      3
 6  5  e     B      4
 7  6  f   NaN      4
 8  7  g   NaN      5

I want to remove all rows which are duplicates with regards to column 'A' 'B'. I want to remove the entry which has a NaN entry (I know that for all dulicates there will be a NaN and a not-NaN entry). The end results should look like this

    A  B Col_1  Col_2
 0  1  a   NaN      2
 1  2  b     A      2
 2  3  c     A      3
 4  4  d     B      3
 6  5  e     B      4
 7  6  f   NaN      4
 8  7  g   NaN      5

All efficient, one-liners are most welcome

Answer 1

If the goal is to only drop the NaN duplicates, a slightly more involved solution is needed.

First, sort on A, B, and Col_1, so NaNs are moved to the bottom for each group. Then call df.drop_duplicates with keep=first:

out = df.sort_values(['A', 'B', 'Col_1']).drop_duplicates(['A', 'B'], keep='first')
print(out)

   A  B Col_1  Col_2
0  1  a   NaN      2
1  2  b     A      2
2  3  c     A      3
4  4  d     B      3
6  5  e     B      4
7  6  f   NaN      4
8  7  g   NaN      5