%!TEX root = manual.tex
%
\chapter{Pickles}
\label{chap:pickles}

One of the most important features of \asdlgen{} is that it automatically
produces functions that can read and write the data structures it generates
to and from a platform and language independent external
representation.
This process of converting data structures in memory into a
sequence of bytes on the disk is referred to as \emph{pickling}.
Since it is possible to generate data structures and pickling code for
any of the supported languages from a single \asdl{} specification,
\asdlgen{} provides an easy
and efficient way to share complex data structures among these languages.

The \asdl{} pickle format requires that both the reader and writer
of the pickler agree on the type of the pickle.
Other than constructor tags for sum types, there is no explicit type
information in the pickle.
In the case of an error the behavior is undefined.
It is also important that the pickling/unpickling to/from files, that the
files be opened in binary mode to prevent line feed translations from corrupting
the pickle.

\section{Binary Pickle Format}

Since \asdl{} data structures have a tree-like form, they can be represented
linearly with a simple prefix encoding.

\subsection{Primitive types}

\begin{description}
  \item[bool] \mbox{}\\
    Boolean values are represented by \lstinline!0! (false) or \lstinline!1! (true)
    and are encoded in one byte.

  \item[int] \mbox{}\\
    The \lstinline!int! type provides 30-bits of signed precision encoded in one to
    four bytes.
    The top two bits of the first byte (bit~6--7) specify the number of additional
    bytes in the encoding and bit~5 specifies the sign of the number.
    Thus values in the range -32 to 31 can be encoded in one byte,
    -8192 to 8191 in two bytes, \etc{}
    A negative number $n$ is represented as the positive number $-(n+1)$.

  \item[uint] \mbox{}\\
    The \lstinline!uint! type provides 30-bits of unsigned precision encoded in one
    to four bytes.
    As with the \lstinline!int! type, the top two bits of the first byte specify the
    number of additional bytes in the encoding.
    Thus values in the range 0 to 63 can be encoded in one byte,
    0 to 16383 unsigned in two bytes, \etc{}

  \item[integer] \mbox{}\\
    The \asdl{} \lstinline!integer! type is represented with a variable-length,
    big-endian, signed-magnitude encoding.
    The high bit of each byte indicates if the byte is the last byte of
    the encoding.
    The bit~6 of the most significant byte is used to determine the
    sign of the value.
    Thus, numbers in the range of -63 to 63 are encoded in one byte.
    Numbers outside of this range require an extra byte for every seven bits
    of precision required.

  \item[string] \mbox{}\\
    Strings are represented with a length-header that describe how many more
    8-bit bytes follow for the string and then the data for the string in bytes.
    The length-header is encoded as a \lstinline!uint! value, thus strings are limited
    to 1,073,741,823 characters.

  \item[identifier] \mbox{}\\
    Identifiers are represented as if they were strings.
\end{description}%

\subsection{Product types}
The fields of a product type are encoded sequentially (left to right)
without any initial tag.

\begin{description}
  \item[sequence types] \mbox{}\\
    Sequence types are represented with an integer length-header followed by
    that many values of that type.
    The length-header is encoded as a \lstinline!uint! value, thus sequences are limited
    to at most 1,073,741,823 items.

  \item[option types] \mbox{}\\
    The encoding of optional values depends on the base type.
    For sum types with more than one constructor, the special tag value of zero is
    used to denote an empty value and non-zero values are interpreted as
    the constructor's tag.
    For any other base type, there is an initial byte that is either one or zero.
    A zero indicates that the value is empty and no more data follows.
    A one indicates that the next value is the value of the optional value.
\end{description}%

\subsection{Enumeration types}

\subsection{Sum types}
Sum types begin with a unique tag to identify the constructor
followed by the fields of the constructor.
The tag is encoded as either one (\lstinline!tag8!) or two (\lstinline!tag16!)
bytes, depending on the number of constructors in the type.
Tag values are assigned in order of constructor definition starting from one (the
value zero is used to encode empty option values).
Fields are packed left to right based of the order in the definition.
If there are any attribute values associated with the type, they are packed left to right
after the tag but before other constructor fields.
    
There are two special cases for sum-type encodings.
If the sum types is an enumeration with only one constructor, then no space is used to
encode the value, and if the type has a single constructor with fields, then it is
encoded without a tag (\ie{}, like a product type).


\subsection{Alias types}

\subsection{primitive types}
User-defined primitive types are pickled/unpickled by user-provided functions (see
\secref{sec:primitive-syntax}).

\section{S-expression Format}
It is also possible to generate a text-based representation of pickles in S-Expression
syntax.

\begin{description}
  \item[bool] \mbox{}\\

  \item[numbers] \mbox{}\\
    The ASDL numeric types (\lstinline!int!, \lstinline!uint!, and \lstinline!integer!)
    are represented by decimal literals.

  \item[string and identifier] \mbox{}\\
    These values are represented by string literals.

  \item[enumeration types] \mbox{}\\
    
  \item[sum types] \mbox{}\\

  \item[sequence types] \mbox{}\\

  \item[option types] \mbox{}\\

  \item[primitive types] \mbox{}\\
    User-defined primitive types are not yet supported in S-Expression form.
    
\end{description}%

\section{XML Pickle Format}

