proxy - Python urllib2: Cannot assign requested address

Question

I am sending thousands of requests using urllib2 with proxies. I have received many of the following error on execution:

urlopen error [Errno 99] Cannot assign requested address

I read here that it may be due to a socket already being bonded. Is that the case? Any suggestions on how to fix this?

Answer 1

Here is an answer to a similar looking question that I prepared earlier.... much earlier... Socket in use error when reusing sockets

The error is different, but the underlying problem is probably the same: you are consuming all available ports and trying to reuse them before the TIME_WAIT state has ended.

[EDIT: in response to comments]

If it is within the capability/spec for your application, one obvious strategy is to control the rate of connections to avoid this situation.

Alternatively, you could use the httplib module. httplib.HTTPConnection() lets you specify a source_address tuple with which you can specify the port from which to make the connection, e.g. this will connect to localhost:1234 from localhost:9999:

import httplib
conn = httplib.HTTPConnection('localhost:1234', source_address=('localhost',9999))
conn.request('GET', '/index.html')

Then it is a matter of managing the source port assignment as described in my earlier answer. If you are on Windows you can use this method to get around the default range of ports 1024-5000.

There is (of course), an upper limit to how many connections you are going to be able to make and it is questionable what sort of an application would require making thousands of connections in rapid succession.