python - How does unicodedata.normalize(form, unistr) work?

Question

On the API doc, http://docs.python.org/2/library/unicodedata.html#unicodedata.normalize. It says

Return the normal form form for the Unicode string unistr. Valid values for form are ‘NFC’, ‘NFKC’, ‘NFD’, and ‘NFKD’.`

The documentation is rather vague, can someone explain the valid values with some examples?

Answer 1

I find the documentation pretty clear, but here are a few code examples:

from unicodedata import normalize

print '%r' % normalize('NFD', u'u00C7')  # decompose: convert ? to "C + ?"
print '%r' % normalize('NFC', u'Cu0327') # compose: convert "C + ?" to ?

Both 'D' (=decompose) forms convert a single combined character (like ?) into two characters (a + two dots). Both 'C' (=compose) forms do the reverse.

The two "K" forms are used to convert characters added to Unicode for compatibility purposes. For example, to support software that cannot draw circles around symbols, there is a set of "circled numbers", like ① (unicode number 2460). When we apply the canonical decomposition (NFD) to it, it doesn't do anything:

print '%r' % normalize('NFD', u'u2460')     # u'u2460'

However, the compatibility decomposition (NFKD) will return the corresponding "compatible" character:

print '%r' % normalize('NFKD', u'u2460')    # 1

See http://en.wikipedia.org/wiki/Unicode_equivalence for more details.