python - UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 23: ordinal not in range(128)

Question

when I try to concatenate this, I get the UnicodeDecodeError when the field contains '?' or '′'. If the field that contains the '?' or '′' is the last I get no error.

#...

nombre = fabrica
nombre = nombre.encode("utf-8") + '-' + sector.encode("utf-8")
nombre = nombre.encode("utf-8") + '-' + unidad.encode("utf-8")

#...

return nombre 

any idea? Many thanks!

Answer 1

You are encoding to UTF-8, then re-encoding to UTF-8. Python can only do this if it first decodes again to Unicode, but it has to use the default ASCII codec:

>>> u'?'
u'xf1'
>>> u'?'.encode('utf8')
'xc3xb1'
>>> u'?'.encode('utf8').encode('utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Don't keep encoding; leave encoding to UTF-8 to the last possible moment instead. Concatenate Unicode values instead.

You can use str.join() (or, rather, unicode.join()) here to concatenate the three values with dashes in between:

nombre = u'-'.join(fabrica, sector, unidad)
return nombre.encode('utf-8')

but even encoding here might be too early.

Rule of thumb: decode the moment you receive the value (if not Unicode values supplied by an API already), encode only when you have to (if the destination API does not handle Unicode values directly).