python - How to debug urllib2 request that uses a basic authentication handler

Question

I'm making a request using urllib2 and the HTTPBasicAuthHandler like so:

import urllib2

theurl = 'http://someurl.com'
username = 'username'
password = 'password'

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, theurl, username, password)

authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)

params = "foo=bar"

response = urllib2.urlopen('http://someurl.com/somescript.cgi', params)

print response.info()

I'm currently getting a httplib.BadStatusLine exception when running this code. How could I go about debugging? Is there a way to see what the raw response is regardless of the unrecognized HTTP status code?

Answer 1

Have you tried setting the debug level in your own HTTP handler? Change your code to something like this:

>>> import urllib2
>>> handler=urllib2.HTTPHandler(debuglevel=1)
>>> opener = urllib2.build_opener(handler)
>>> urllib2.install_opener(opener)
>>> resp=urllib2.urlopen('http://www.google.com').read()
send: 'GET / HTTP/1.1
      Accept-Encoding: identity
      Host: www.google.com
      Connection: close
      User-Agent: Python-urllib/2.7'
reply: 'HTTP/1.1 200 OK'
header: Date: Sat, 08 Oct 2011 17:25:52 GMT
header: Expires: -1
header: Cache-Control: private, max-age=0
header: Content-Type: text/html; charset=ISO-8859-1
... the remainder of the send / reply other than the data itself 

So the three lines to prepend are:

handler=urllib2.HTTPHandler(debuglevel=1)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
... the rest of your urllib2 code...

That will show the raw HTTP send / reply cycle on stderr.

Edit from comment

Does this work?

... same code as above this line
opener=urllib2.build_opener(authhandler, urllib2.HTTPHandler(debuglevel=1))
... rest of your code