The UTF8 structure
Author: John Reppy
Keywords: Standard ML, SML, SML/NJ Library

The SML of NJ Library
The `UTF8` structure

The `UTF8` structure provides support for working with UTF-8 encoded strings. UTF-8 is a way to represent Unicode code points in an 8-bit character type while being backward compatible with the ASCII encoding for 7-bit characters. The encoding scheme uses one to four bytes as follows:

Wide Character Bits | Byte 0 | Byte 1 | Byte 2 | Byte 3
--- | --- | --- | --- | ---
`00000` `00000000` `0xxxxxxx` | `0xxxxxxx` |  |  |
`00000` `00000yyy` `yyxxxxxx` | `110yyyyy` | `10xxxxxx` |  |
`00000` `zzzzyyyy` `yyxxxxxx` | `1110zzzz` | `10yyyyyy` | `10xxxxxx` |
`wwwzz` `zzzzyyyy` `yyxxxxxx` | `11110www` | `10zzzzzz` | `10yyyyyy` | `10xxxxxx`

## Synopsis

    signature UTF8
    structure UTF8 :> UTF8

## Interface

    type wchar = word

    val maxCodePoint : wchar

    exception Incomplete

    val getu : (char, 'strm) StringCvt.reader -> (wchar, 'strm) StringCvt.reader

    val encode : wchar -> string

    val isAscii : wchar -> bool
    val toAscii : wchar -> char
    val fromAscii : char -> wchar

    val toString : wchar -> string

    val size : string -> int

    val explode : string -> wchar list
    val implode : wchar list -> string

    val map : (wchar -> wchar) -> string -> string
    val app : (wchar -> unit) -> string -> unit
    val fold : ((wchar * 'a) -> 'a) -> 'a -> string -> 'a
    val all : (wchar -> bool) -> string -> bool
    val exists : (wchar -> bool) -> string -> bool

## Description

`type wchar = word`

  The type of a Unicode code point.
  Note that we use the `word` type for this because SML/NJ does not currently have a wide-character type. If such a type is introduced, then this type definition will likely change.

`val maxCodePoint : wchar`

  The maximum code point in the Unicode character set (`0wx10FFFF`).

`exception Incomplete`

  This exception is raised when certain operations are applied to incomplete strings (i.e., strings that end with a partial UTF-8 character encoding).

`val getu : (char, 'strm) StringCvt.reader -> (wchar, 'strm) StringCvt.reader`

  `getu getc` returns a wide-character reader for the character reader `getc`. The resulting reader will raise the `Incomplete` exception if it encounters an incomplete UTF-8 character.

`val encode : wchar -> string`

  `encode wc` returns the UTF-8 encoding of the wide character `wc`. This expression raises the `Domain` exception if `wc` is greater than the maximum Unicode code point.

`val isAscii : wchar -> bool`

  `isAscii wc` returns `true` if, and only if, `wc` is an ASCII character.

`val toAscii : wchar -> char (* truncates to 7-bits *)`

  `toAscii wc` converts `wc` to an 8-bit character by truncating `wc` to its low seven bits.

`val fromAscii : char -> wchar (* truncates to 7-bits *)`

  `toAscii c` converts the 8-bit character `c` to a wide character in the ASCII range (the high bit of `c` is ignored).

`val toString : wchar -> string`

  `toString wc` returns a printable string representation of a wide character as a Unicode escape sequence.

`val size : string -> int`

  `size s` returns the number of UTF-8 encoded Unicode characters in the string `s`. This expression raises the `Incomplete` exception if an incomplete character is encountered.

`val explode : string -> wchar list`

  `explode s` returns the list of UTF-8 encoded Unicode characters that comprise the string `s`.

`val implode : wchar list -> string`

  `implode wcs` returns the UTF-8 encoded string that represents the list `wcs` of Unicode code points. This expression raises the `Domain` exception if any character in the list is greater than the maximum Unicode code point.

`val map : (wchar -> wchar) -> string -> string`

  `map f s` maps the function `f` over the UTF-8 encoded characters in the string `s` to produce a new UTF-8 string. This expression raises the `Incomplete` exception if an incomplete character is encountered and the `Domain` exception if `f` returns a value that is greater than the maximum Unicode code point. It is equivalent to the expression

      implode (List.map f (explode s))

`val app : (wchar -> unit) -> string -> unit`

  `app f s` applies the function `f` to the UTF-8 encoded characters in the string `s`. This expression raises the `Incomplete` exception if an incomplete character is encountered. It is equivalent to the expression

      List.app f (explode s)

`val fold : ((wchar * 'a) -> 'a) -> 'a -> string -> 'a`

  `fold f init s` folds a function from left-to-right over the UTF-8 encoded characters in the string. This expression raises the `Incomplete` exception if an incomplete character is encountered. It is equivalent to the expression

      List.foldl f init (explode s)

`val all : (wchar -> bool) -> string -> bool`

  `all pred s` returns `true` if, and only if, the function `pred` returns true for all of the UTF-8 encoded characters in the string. It short-circuits evaluation as soon as a character is encountered for which `pred` returns `false`. This expression raises the `Incomplete` exception if an incomplete character is encountered. It is equivalent to the expression

      List.all pred (explode s)

  when `s` only contains complete characters.

`val exists : (wchar -> bool) -> string -> bool`

  `exists pred s` returns `true` if, and only if, the function `pred` returns `true` for at least one UTF-8 encoded character in the string `s`. It short-circuits evaluation as soon as a character is encountered for which `pred` returns `true`. This expression raises the `Incomplete` exception if an incomplete character is encountered. It is equivalent to the expression

      List.exists pred (explode s)

  when `s` only contains complete characters.

## See Also

The Util Library

SML/NJ Version 110.97 (July 16, 2020)
Last updated 2020-04-19 22:12:26 UTC