python - 'ascii' codec error in beautifulsoup

Question

I am using beautifulsoup for scraping data from the html page. Till yesterday every thing was fine. But Now i am getting the error:

'ascii' codec can't encode character u'xa9' in position 86700: ordinal not in range(128)

i am using the code:

import urllib2
from BeautifulSoup import BeautifulSoup

page = urllib2.urlopen(url).read()
soup = BeautifulSoup(page)

This is giving me the error.

Answer 1

A wild guess:

Try specifying the encoding of the page?

soup = BeautifulSoup(page, fromEncoding=<encoding of the page>)

This can also be a problem with the Python installation. If you print non-ASCII characters without BeautifulSoup, do you face the same problem? If yes, then you need to set the encoding:

import sys
sys.setdefaultencoding("utf-8") # or whatever you want the default encoding to be.