Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.2k views
in Technique[技术] by (71.8m points)

unicode - Python : UnicodeEncodeError: 'latin-1' codec can't encode character

I am at a scenario where I call api and based on the results from api I call database for each record that I in api. My api call return strings and when I make the database call for the items return by api, for some elements I get the following error.

Traceback (most recent call last):
  File "TopLevelCategories.py", line 267, in <module>
    cursor.execute(categoryQuery, {'title': startCategory});
  File "/opt/ts/python/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py", line 158, in execute
    query = query % db.literal(args)
  File "/opt/ts/python/2.7/lib/python2.7/site-packages/MySQLdb/connections.py", line 265, in literal
    return self.escape(o, self.encoders)
  File "/opt/ts/python/2.7/lib/python2.7/site-packages/MySQLdb/connections.py", line 203, in unicode_literal
    return db.literal(u.encode(unicode_literal.charset))
UnicodeEncodeError: 'latin-1' codec can't encode character u'u2013' in position 3: ordinal not in range(256)

The segment of my code the above error is referring is:

         ...    
         for startCategory in value[0]:
            categoryResults = []
            try:
                categoryRow = ""
                baseCategoryTree[startCategory] = []
                #print categoryQuery % {'title': startCategory}; 
                cursor.execute(categoryQuery, {'title': startCategory}) #unicode issue
                done = False
                cont...

After doing some google search I tried the following on my command line to understand whats going on...

>>> import sys
>>> u'u2013'.encode('iso-8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't encode character u'u2013' in position 0: ordinal not in range(256)
>>> u'u2013'.encode('cp1252')
'x96'
>>> 'u2013'.encode('cp1252')
'\u2013'
>>> u'u2013'.encode('cp1252')
'x96'

But I am not sure what would be the solution to overcome this issue. Also I don't know what is the theory behind encode('cp1252') it would be great if I can get some explanation on what I tried above.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

If you need Latin-1 encoding, you have several options to get rid of the en-dash or other code points above 255 (characters not included in Latin-1):

>>> u = u'hellou2013world'
>>> u.encode('latin-1', 'replace')    # replace it with a question mark
'hello?world'
>>> u.encode('latin-1', 'ignore')     # ignore it
'helloworld'

Or do your own custom replacements:

>>> u.replace(u'u2013', '-').encode('latin-1')
'hello-world'

If you aren't required to output Latin-1, then UTF-8 is a common and preferred choice. It is recommended by the W3C and nicely encodes all Unicode code points:

>>> u.encode('utf-8')
'helloxe2x80x93world'

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...