python - Pandas combining sparse columns in dataframe

Question

I am using Python, Pandas for data analysis. I have sparsely distributed data in different columns like following

| id | col1a | col1b | col2a | col2b | col3a | col3b |
|----|-------|-------|-------|-------|-------|-------|
|  1 |   11  |   12  |  NaN  |  NaN  |  NaN  |  NaN  |
|  2 |  NaN  |  NaN  |   21  |   86  |  NaN  |  NaN  |
|  3 |   22  |   87  |  NaN  |  NaN  |  NaN  |  NaN  |
|  4 |  NaN  |  NaN  |   NaN |  NaN  |  545  |   32  |

I want to combine this sparsely distributed data in different columns to tightly packed column like following.

| id | group |  cola |  colb |
|----|-------|-------|-------|
| 1  |  g1   |   11  |   12  |
| 2  |  g2   |   21  |   86  |
| 3  |  g1   |   22  |   87  |
| 4  |  g3   |  545  |   32  |

What I have tried is doing following, but not able to do it properly

df['cola']=np.nan
df['colb']=np.nan
df['cola'].fillna(df.col1a,inplace=True)
df['colb'].fillna(df.col1b,inplace=True)
df['cola'].fillna(df.col2a,inplace=True)
df['colb'].fillna(df.col2b,inplace=True)
df['cola'].fillna(df.col3a,inplace=True)
df['colb'].fillna(df.col3b,inplace=True)

But I think there must be more concise and efficient way way of doing this. How to do this in better way?

Answer 1

You can use df.stack() assuming 'id' is your index else set 'id' as index. Then use pd.pivot_table.

df = df.stack().reset_index(name='val',level=1)
df['group'] = 'g'+ df['level_1'].str.extract('col(d+)')
df['level_1'] = df['level_1'].str.replace('col(d+)','')
df.pivot_table(index=['id','group'],columns='level_1',values='val')

level_1    cola  colb
id group
1  g1      11.0  12.0
2  g2      21.0  86.0
3  g1      22.0  87.0
4  g3     545.0  32.0