python - Removing u2018 and u2019 character

Question

I am using Beautiful Soup to parse webpages and printing the name of the webpages visited on the terminal. However, often the name of the webpage has single left (u2018) and right(u2019) character which the python can't print as it gives charmap encoding error. Is there any way to remove these characters?

Answer 1

These codes are Unicode for the single left and right quote characters. You can replace them with their ASCII equivalent which Python shouldn't have any problem printing on your system:

>>> print u"u2018Hiu2019"
‘Hi’
>>> print u"u2018Hiu2019".replace(u"u2018", "'").replace(u"u2019", "'")
'Hi'

Alternatively with regex:

import re
s = u"u2018Hiu2019"
>>> print re.sub(u"(u2018|u2019)", "'", s)
'Hi'

However Python shouldn't have any problem printing the Unicode version of these as well. It's possible that you are using str() somewhere which will try to convert your unicode to ascii and throw your exception.