Character Encoding and Unicode

Character encoding is a rule for representing text as bytes. A byte contains eight bits. Unicode assigns code points to characters; an encoding such as UTF-8 maps Unicode scalar values to byte sequences. A code point is not a byte, and a displayed character can contain more than one code point.

For example, A has code point U+0041 and uses one byte in UTF-8. The precomposed character é has code point U+00E9 and uses the two bytes C3 A9. The visually similar sequence e followed by a combining acute accent uses two code points and three UTF-8 bytes. A visible-character limit therefore differs from a code-point or byte limit; normalization is another contract choice, not an automatic consequence of using Unicode.

Encoding converts text to bytes; decoding interprets bytes as text. Reading the bytes C3 A9 as Windows-1252 produces é instead of é. This garbled text is often called mojibake. A wrong encoding can also cause an error, or appear harmless when the input happens to use only shared characters. Successful decoding is not evidence that the encoding was correct.

Use the producer’s declared encoding and preserve original bytes when repair matters. This specific mojibake can be reversed if every byte survived the mistaken conversion. Replacement characters, discarded bytes, or substituted question marks can remove information, making the original text unrecoverable from the damaged string alone. Check the source rather than repeatedly converting until text looks plausible.

Reference: Python Unicode HOWTO. For worked storage examples, see How Computers Store Data.