python - UnicodeEncodeError: 'ascii' codec can't encode character u'xe4'

Question

I permanently get the following error:

UnicodeEncodeError: 'ascii' codec can't encode character u'xe4' in position 27: ordinal not in range(128)

I already tried

1. x.encode("ascii", "ignore")
2. x.encode("utf-8")
3. x.decode("utf-8")

However, nothing works.

Answer 1

You have to discover in which encoding is this character at the source.

I guess this is ISO-8859-1 (european languages), in which case it's "?", but you should check. It could also be cyrillic or greek.

See http://en.wikipedia.org/wiki/ISO/IEC_8859-1 for a complete list of characters in this encoding.

Using this information, you can ask Python to convert it :

In Python 2.7

>>> s = 'xe4'
>>> t = s.decode('iso-8859-1')
>>> print t
?
>>> for c in t:
...   print ord(c)
...
228
>>> u = t.encode('utf-8')
>>> print u
?
>>> for c in bytes(u):
...   print ord(c)
...
195
164

String t is internally encoded in ISO-8859-1 in Python. String u is internally encoded in UTF-8, and that character takes 2 bytes in UTF-8. Notice also that the print instruction "knows" how to display these different encodings.