python - Getting the regression line to plot from a Pandas regression

Question

I have tried with both the (pandas)pd.ols and the (statsmodels)sm.ols to get a regression scatter plot with the regression line, I can get the scatter plot but I can't seem to get the parameters to get the regression line to plot. It is probably obvious that I am doing some cut and paste coding here :-( (using this as a guide: http://nbviewer.ipython.org/github/weecology/progbio/blob/master/ipynbs/statistics.ipynb

My data is in a pandas DataFrame and the x column is merged2[:-1].lastqu and the y data column is merged2[:-1].Units My code is now as follows: to get the regression:

def fit_line2(x, y):
    X = sm.add_constant(x, prepend=True) #Add a column of ones to allow the calculation of the intercept
    model = sm.OLS(y, X,missing='drop').fit()
    """Return slope, intercept of best fit line."""
    X = sm.add_constant(x)
    return model
model=fit_line2(merged2[:-1].lastqu,merged2[:-1].Units)
print fit.summary()

^^^^ seems ok

intercept, slope = model.params  << I don't think this is quite right
plt.plot(merged2[:-1].lastqu,merged2[:-1].Units, 'bo')
plt.hold(True)

^^^^^ this gets the scatter plot done ****and the below does not get me a regression line

x = np.array([min(merged2[:-1].lastqu), max(merged2[:-1].lastqu)])
y = intercept + slope * x
plt.plot(x, y, 'r-')
plt.show()

A snippit of the Dataframe: the [:-1] eliminates the current period from the data which will subsequently be a projection

Units   lastqu  Uperchg lqperchg    fcast   errpercent  nfcast
date                            
2000-12-31   7177    NaN     NaN     NaN     NaN     NaN     NaN
2001-12-31   10694   2195.000000     0.490038    NaN     10658.719019    1.003310    NaN
2002-12-31   11725   2469.000000

Edit:

I found I could do:

fig = plt.figure(figsize=(12,8))
fig = sm.graphics.plot_regress_exog(model, "lastqu", fig=fig)

as described here in the Statsmodels doc which seems to get the main thing I wanted (and more) I'd still like to know where I went wrong in the prior code!

Answer 1

Check what values you have in your arrays and variables.

My guess is that your x is just nans, because you use Python's min and max. At least that happens with the version of Pandas that I have currently open.

The min and max methods should work, since they know how to handle nans or missing values

>>> x = pd.Series([np.nan,2], index=['const','slope'])
>>> x
const   NaN
slope     2
dtype: float64

>>> min(x)
nan
>>> max(x)
nan

>>> x.min()
2.0
>>> x.max()
2.0