UTF(6)UTF(6)
NAME
UTF, Unicode, ASCII, rune – character set and format
DESCRIPTION
The Plan 9 character set and representation are
based on the Unicode Standard and on the ISO multibyte
UTF-8
encoding (Universal Character
Set Transformation Format, 8 bits wide).
The Unicode Standard represents its characters in 21
bits;
UTF-8
represents such
values in an 8-bit byte stream.
Throughout this manual,
UTF-8
is shortened to
UTF.
In Plan 9, a
rune
is a 32-bit quantity representing a Unicode character.
Internally, programs may store characters as runes.
However, any external manifestation of textual information,
in files or at the interface between programs, uses a
machine-independent, byte-stream encoding called
UTF.
UTF
is designed so the 7-bit
ASCII
set (values hexadecimal 00 to 7F),
appear only as themselves
in the encoding.
Runes with values above 7F appear as sequences of two or more
bytes with values only from 80 to FF.
The
UTF
encoding of the Unicode Standard is backward compatible with
ASCII\c
:
programs presented only with
ASCII
work on Plan 9
even if not written to deal with
UTF,
as do
programs that deal with uninterpreted byte streams.
However, programs that perform semantic processing on
ASCII
graphic
characters must convert from
UTF
to runes
in order to work properly with non-\c
ASCII
input.
See
rune(2).
Letting numbers be binary,
a rune x is converted to a multibyte
UTF
sequence
as follows:
001. x in [00000000.00000000.0bbbbbbb] → 0bbbbbbb
010. x in [00000000.00000bbb.bbbbbbbb] → 110bbbbb, 10bbbbbb
011. x in [00000000.bbbbbbbb.bbbbbbbb] → 1110bbbb, 10bbbbbb, 10bbbbbb
100. x in [000bbbbb.bbbbbbbb.bbbbbbbb] → 11110bbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
Conversion 001 provides a one-byte sequence that spans the
ASCII
character set in a compatible way.
Conversions 010, 011 and 100 represent higher-valued characters
as sequences of two, three or four bytes with the high bit set.
Plan 9 does not support the 5 and 6 byte sequences proposed by X-Open.
When there are multiple ways to encode a value, for example rune 0,
the shortest encoding is used.
In the inverse mapping,
any sequence except those described above
is incorrect and is converted to rune hexadecimal FFFD.
FILES
/lib/unicode
table of characters and descriptions, suitable for
look(1).
SEE ALSO
ascii(1),
tcs(1),
rune(2),
keyboard(6),
The Unicode Standard .