python - Calculate correlation between all columns of a DataFrame and all columns of another DataFrame?

Question

I have a DataFrame object stocks filled with stock returns. I have another DataFrame object industries filled with industry returns. I want to find each stock's correlation with each industry.

import numpy as np
np.random.seed(123)

df1=pd.DataFrame( {'s1':np.random.randn(10000), 's2':np.random.randn(10000) } )
df2=pd.DataFrame( {'i1':np.random.randn(10000), 'i2':np.random.randn(10000) } )

The expensive way to do this is to merge the two DataFrame objects, calculate correlation, and then throw out all the stock to stock and industry to industry correlations. Is there a more efficient way to do this?

Answer 1

And here's a one-liner that uses apply on the columns and avoids the nested for loops. The main benefit is that apply builds the result in a DataFrame.

df1.apply(lambda s: df2.corrwith(s))