python - How do I compare a Unicode string that has different bytes, but the same value?

Question

I'm comparing Unicode strings between JSON objects.

They have the same value:

a = '人口じんこうに膾炙かいしゃする'
b = '人口じんこうに膾炙かいしゃする'

But they have different Unicode representations:

String a : u'u4ebau53e3u3058u3093u3053u3046u306bu81beu7099u304bu3044u3057u3083u3059u308b'
String b : u'u4ebau53e3u3058u3093u3053u3046u306bu81beuf9fbu304bu3044u3057u3083u3059u308b'

How can I compare between two Unicode strings on their value?

Answer 1

Unicode normalization will get you there for this one:

>>> import unicodedata
>>> unicodedata.normalize("NFC", "uf9fb") == "u7099"
True

Use unicodedata.normalize on both of your strings before comparing them with == to check for canonical Unicode equivalence.

Character U+F9FB is a "CJK Compatibility" character. These characters decompose into one or more regular CJK characters when normalized.