Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.8k views
in Technique[技术] by (71.8m points)

character encoding - Python: Converting from ISO-8859-1/latin1 to UTF-8

I have this string that has been decoded from Quoted-printable to ISO-8859-1 with the email module. This gives me strings like "xC4pple" which would correspond to "?pple" (Apple in Swedish). However, I can't convert those strings to UTF-8.

>>> apple = "xC4pple"
>>> apple
'xc4pple'
>>> apple.encode("UTF-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in     range(128)

What should I do?

Question&Answers:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

This is a common problem, so here's a relatively thorough illustration.

For non-unicode strings (i.e. those without u prefix like u'xc4pple'), one must decode from the native encoding (iso8859-1/latin1, unless modified with the enigmatic sys.setdefaultencoding function) to unicode, then encode to a character set that can display the characters you wish, in this case I'd recommend UTF-8.

First, here is a handy utility function that'll help illuminate the patterns of Python 2.7 string and unicode:

>>> def tell_me_about(s): return (type(s), s)

A plain string

>>> v = "xC4pple" # iso-8859-1 aka latin1 encoded string

>>> tell_me_about(v)
(<type 'str'>, 'xc4pple')

>>> v
'xc4pple'        # representation in memory

>>> print v
?pple             # map the iso-8859-1 in-memory to iso-8859-1 chars
                  # note that 'xc4' has no representation in iso-8859-1, 
                  # so is printed as "?".

Decoding a iso8859-1 string - convert plain string to unicode

>>> uv = v.decode("iso-8859-1")
>>> uv
u'xc4pple'       # decoding iso-8859-1 becomes unicode, in memory

>>> tell_me_about(uv)
(<type 'unicode'>, u'xc4pple')

>>> print v.decode("iso-8859-1")
?pple             # convert unicode to the default character set
                  # (utf-8, based on sys.stdout.encoding)

>>> v.decode('iso-8859-1') == u'xc4pple'
True              # one could have just used a unicode representation 
                  # from the start

A little more illustration — with “?”

>>> u"?" == u"xc4"
True              # the native unicode char and escaped versions are the same

>>> "?" == u"xc4"  
False             # the native unicode char is 'xc3x84' in latin1

>>> "?".decode('utf8') == u"xc4"
True              # one can decode the string to get unicode

>>> "?" == "xc4"
False             # the native character and the escaped string are
                  # of course not equal ('xc3x84' != 'xc4').

Encoding to UTF

>>> u8 = v.decode("iso-8859-1").encode("utf-8")
>>> u8
'xc3x84pple'    # convert iso-8859-1 to unicode to utf-8

>>> tell_me_about(u8)
(<type 'str'>, 'xc3x84pple')

>>> u16 = v.decode('iso-8859-1').encode('utf-16')
>>> tell_me_about(u16)
(<type 'str'>, 'xffxfexc4x00px00px00lx00ex00')

>>> tell_me_about(u8.decode('utf8'))
(<type 'unicode'>, u'xc4pple')

>>> tell_me_about(u16.decode('utf16'))
(<type 'unicode'>, u'xc4pple')

Relationship between unicode and UTF and latin1

>>> print u8
?pple             # printing utf-8 - because of the encoding we now know
                  # how to print the characters

>>> print u8.decode('utf-8') # printing unicode
?pple

>>> print u16     # printing 'bytes' of u16
???pple

>>> print u16.decode('utf16')
?pple             # printing unicode

>>> v == u8
False             # v is a iso8859-1 string; u8 is a utf-8 string

>>> v.decode('iso8859-1') == u8
False             # v.decode(...) returns unicode

>>> u8.decode('utf-8') == v.decode('latin1') == u16.decode('utf-16')
True              # all decode to the same unicode memory representation
                  # (latin1 is iso-8859-1)

Unicode Exceptions

 >>> u8.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0:
  ordinal not in range(128)

>>> u16.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0:
  ordinal not in range(128)

>>> v.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0:
  ordinal not in range(128)

One would get around these by converting from the specific encoding (latin-1, utf8, utf16) to unicode e.g. u8.decode('utf8').encode('latin1').

So perhaps one could draw the following principles and generalizations:

  • a type str is a set of bytes, which may have one of a number of encodings such as Latin-1, UTF-8, and UTF-16
  • a type unicode is a set of bytes that can be converted to any number of encodings, most commonly UTF-8 and latin-1 (iso8859-1)
  • the print command has its own logic for encoding, set to sys.stdout.encoding and defaulting to UTF-8
  • One must decode a str to unicode before converting to another encoding.

Of course, all of this changes in Python 3.x.

Hope that is illuminating.

Further reading

And the very illustrative rants by Armin Ronacher:


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...