\lstset{language=transcript}

\section{Introduction: How Cylc Works}
\label{HowCylcWorks}

\subsection{Scheduling Forecast Suites}
\label{SchedulingForecastSuites}

Environmental forecasting suites generate forecast products from a
potentially large group of interdependent scientific models and
associated data processing tasks. They are constrained by availability
of external driving data: typically one or more tasks will wait on real
time observations and/or model data from an external system, and these
will drive other downstream tasks, and so on. The dependency diagram for
a single forecast cycle point in such a system is a {\em Directed Acyclic
Graph} as shown in Figure~\ref{fig-dep-one} (in our terminology, a {\em
forecast cycle point} is comprised of all tasks with a common {\em cycle
point}, which is the nominal analysis time or start time of the forecast
models in the group). In real time operation processing will consist of
a series of distinct forecast cycle points that are each initiated, after a
gap, by arrival of the new cycle point's external driving data.

From a job scheduling perspective task execution order in such a system
must be carefully controlled in order to avoid dependency violations.
Ideally, each task should be queued for execution at the instant its
last prerequisite is satisfied; this is the best that can be done even
if queued tasks are not able to execute immediately because of resource
contention.

\subsection{EcoConnect}
\label{EcoConnect}

Cylc was developed for the EcoConnect Forecasting System at NIWA
(National Institute of Water and Atmospheric Research, New Zealand).
EcoConnect takes real time atmospheric and stream flow observations, and
operational global weather forecasts from the Met Office (UK), and uses
these to drive global sea state and regional data assimilating weather
models, which in turn drive regional sea state, storm surge, and
catchment river models, plus tide prediction, and a large number of
associated data collection, quality control, preprocessing,
post-processing, product generation, and archiving tasks.\footnote{Future
plans for EcoConnect include additional deterministic regional weather
forecasts and a statistical ensemble.} The global sea state forecast
runs once daily. The regional weather forecast runs four times daily but
it supplies surface winds and pressure to several downstream models that
run only twice daily, and precipitation accumulations to catchment river
models that run on an hourly cycle assimilating real time stream flow
observations and using the most recently available regional weather
forecast. EcoConnect runs on heterogeneous distributed hardware,
including a massively parallel supercomputer and several Linux servers.


\subsection{Dependence Between Tasks}

\subsubsection{Intra-cycle Dependence}
\label{IntracycleDependence}

Most dependence between tasks applies within a single forecast cycle
point. Figure~\ref{fig-dep-one} shows the dependency diagram for a single
forecast cycle point of a simple example suite of three forecast models
({\em a, b,} and {\em c}) and three post processing or product generation
tasks ({\em d, e} and {\em f}). A scheduler capable of handling this
must manage, within a single forecast cycle point, multiple parallel
streams of execution that branch when one task generates output for
several downstream tasks, and merge when one task takes input from several
upstream tasks.

\begin{figure}
    \begin{center}
        \includegraphics[width=6cm]{graphics/png/orig/dep-one-cycle.png}
    \end{center}
    \caption[A single cycle point dependency graph for a simple suite]
    {\scriptsize
    The dependency graph for a single forecast cycle point of a simple
    example suite. Tasks {\em a, b,} and {\em c} represent forecast models,
    {\em d, e} and {\em f} are post processing or product generation
    tasks, and {\em x} represents external data that the upstream
    forecast model depends on.}
    \label{fig-dep-one}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/timeline-one.png}
    \end{center}
    \caption[A single cycle point job schedule for real time operation]
    {\scriptsize
    The optimal job schedule for two consecutive cycle points of our
    example suite during real time operation, assuming that all tasks
    trigger off upstream tasks finishing completely. The horizontal
    extent of a task bar represents its execution time, and the vertical
    blue lines show when the external driving data becomes available.}
    \label{fig-time-one}
\end{figure}

Figure~\ref{fig-time-one} shows the optimal job schedule for two
consecutive cycle points of the example suite in real time operation, given
execution times represented by the horizontal extent of the task bars.
There is a time gap between cycle points as the suite waits on new external
driving data. Each task in the example suite happens to trigger off
upstream tasks {\em finishing}, rather than off any intermediate output
or event; this is merely a simplification that makes for clearer
diagrams.

\begin{figure}
    \begin{center}
        \includegraphics[width=10cm]{graphics/png/orig/dep-two-cycles-linked.png}
    \end{center}
    \caption[What if the external driving data is available early?]{\scriptsize If
    the external driving data is available in advance, can we start
    running the next cycle point early?}
    \label{fig-dep-two-linked}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=6cm]{graphics/png/orig/timeline-one-c.png}
    \end{center}
    \caption[Attempted overlap of consecutive single-cycle-point job
    schedules]{\scriptsize A naive attempt to overlap two consecutive cycle
    points using the single-cycle-point dependency graph. The red shaded
    tasks will fail because of dependency violations (or will not be able to
    run because of upstream dependency violations).}
    \label{fig-overlap}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/timeline-one-a.png}
    \end{center}
    \caption[The only safe multi-cycle-point job schedule?]
    {\scriptsize The best that can be done {\em in general} when
    inter-cycle dependence is ignored.}
    \label{fig-job-no-overlap}
\end{figure}

Now the question arises, what happens if the external driving data for
upcoming cycle points is available in advance, as it would be after a
significant delay in operations, or when running a historical case
study?  While the forecast model {\em a} appears to depend only on the
external data {\em x} at this stage of the discussion, in fact it would
typically also depend on its own previous instance for the model {\em
background state} used in initializing the new forecast. Thus, as
alluded to in Figure~\ref{fig-dep-two-linked}, task {\em a} could in
principle start
as soon as its predecessor has finished. Figure~\ref{fig-overlap}
shows, however, that starting a whole new cycle point at this point is
dangerous - it results in dependency violations in half of the tasks in
the example suite. In fact the situation could be even worse than this
- imagine that task {\em b} in the first cycle point is delayed for some
reason {\em after} the second cycle point has been launched. Clearly we must
consider handling inter-cycle dependence explicitly or else agree not to
start the next cycle point early, as is illustrated in
Figure~\ref{fig-job-no-overlap}.

\subsubsection{Inter-Cycle Dependence}
\label{InterCyclePointDependence}

Forecast models typically depend on their own most recent previous
forecast for background state or restart files of some kind (this is
called {\em warm cycling}) but there can also be inter-cycle dependence
between different tasks. In an atmospheric forecast analysis suite, for
instance, the weather model may generate background states for observation
processing and data-assimilation tasks in the next cycle point as well as for
the next forecast model run. In real time operation inter-cycle
dependence can be ignored because it is automatically satisfied when one cycle
point finishes before the next begins. If it is not ignored it drastically
complicates the dependency graph by blurring the clean boundary between
cycle points. Figure~\ref{fig-dep-multi} illustrates the problem for our
simple example suite assuming minimal inter-cycle dependence: the warm
cycled models ($a$, $b$, and $c$) each depend on their own previous instances.

For this reason, and because we tend to see forecasting suites in terms of
their real time characteristics, other metaschedulers have ignored
inter-cycle dependence and are thus restricted to running entire cycle
points in sequence at all times. This does not affect normal real time
operation but it can be a serious impediment when advance availability of
external driving data makes it possible, in principle, to run some tasks from
upcoming cycle points before the current cycle point is finished - as was
suggested at the end of the previous section. This can occur, for instance,
after operational delays (late arrival of external data, system maintenance,
etc.) and to an even greater extent in historical case studies and parallel
test suites started behind a real time operation. It can be a serious problem
for suites that have little downtime between forecast cycle points and
therefore take many cycle points to catch up after a delay. Without taking
account of inter-cycle dependence, the best that can be done, in
general, is to reduce the gap between cycle points to zero as shown in
Figure~\ref{fig-job-no-overlap}. A limited crude overlap of the single cycle
point job schedule may be possible for specific task sets but the allowable
overlap may change if new tasks are added, and it is still dangerous: it
amounts to running different parts of a dependent system as if they were not
dependent and as such it cannot be guaranteed that some unforeseen delay in
one cycle point, after the next cycle point has begun, (e.g.\ due to resource
contention or task failures) won't result in dependency violations.

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/dep-multi-cycle.png}
    \end{center}
    \caption[The complete multi-cycle-point dependency graph]
    {\scriptsize The complete dependency graph for the example suite, assuming
    the least possible inter-cycle dependence: the forecast models ($a$,
    $b$, and $c$) depend on their own previous instances. The dashed arrows
    show connections to previous and subsequent forecast cycle points.}
    \label{fig-dep-multi}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=6cm]{graphics/png/orig/timeline-two-cycles-optimal.png}
    \end{center}
    \caption[The optimal two-cycle-point job schedule]
    {\scriptsize The optimal two cycle job schedule when the next cycle's driving data is available in
    advance, possible in principle when inter-cycle dependence is
    handled explicitly.}
    \label{fig-optimal-two}
\end{figure}

Figure~\ref{fig-optimal-two} shows, in contrast to
Figure~\ref{fig-overlap}, the optimal two cycle point job schedule obtained by
respecting all inter-cycle dependence. This assumes no delays due to
resource contention or otherwise - i.e.\ every task runs
as soon as it is ready to run. The scheduler running
this suite must be able to adapt dynamically to external conditions
that impact on multi-cycle-point scheduling in the presence of
inter-cycle dependence or else, again, risk bringing the system down
with dependency violations.

\begin{figure}
    \begin{center}
        \includegraphics[width=12cm]{graphics/png/orig/timeline-three.png}
    \end{center}
    \caption[Comparison of job schedules after a delay]{\scriptsize Job
    schedules for the example suite after a delay of almost one whole
    forecast cycle point, when inter-cycle dependence is
    taken into account (above the time axis), and when it is not
    (below the time axis). The colored lines indicate the time that
    each cycle point is delayed, and normal ``caught up'' cycle points
    are shaded gray.}
    \label{fig-time-three}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/timeline-two.png}
    \end{center}
    \caption[Optimal job schedule when all external data is
    available]{\scriptsize Job schedules for the example suite in case study
    mode, or after a long delay, when the external driving data are
    available many cycle points in advance. Above the time axis is the optimal
    schedule obtained when the suite is constrained only by its true
    dependencies, as in Figure \ref{fig-dep-two-linked}, and underneath
    is the best that can be done, in general, when inter-cycle
    dependence is ignored.}
    \label{fig-time-two}
\end{figure}

To further illustrate the potential benefits of proper inter-cycle
dependency handling, Figure~\ref{fig-time-three} shows an operational
delay of almost one whole cycle point in a suite with little downtime between
cycle points. Above the time axis is the optimal schedule that is possible in
principle when inter-cycle dependence is taken into account, and below
it is the only safe schedule possible {\em in general} when it is ignored.
In the former case, even the cycle point immediately after the delay is hardly
affected, and subsequent cycle points are all on time, whilst in the latter
case it takes five full cycle points to catch up to normal real time
operation.

%Note that simply overlapping the single cycle point schedules of
%Figure~\ref{fig-time-one} from the same start point would have resulted
%in dependency violation by task {\em c}.

Similarly, Figure~\ref{fig-time-two} shows example suite job schedules
for an historical case study, or when catching up after a very long
delay; i.e.\ when the external driving data are available many cycle
points in advance. Task {\em a}, which as the most upstream forecast
model is likely to be a resource intensive atmosphere or ocean model,
has no upstream dependence on co-temporal tasks and can therefore run
continuously, regardless of how much downstream processing is yet to be
completed in its own, or any previous, forecast cycle point (actually,
task {\em a} does depend on co-temporal task {\em x} which waits on the
external driving data, but that returns immediately when the data is
available in advance, so the result stands). The other forecast models
can also cycle continuously or with a short gap between, and some
post processing tasks, which have no previous-instance dependence, can
run continuously or even overlap (e.g.\ {\em e} in this case). Thus,
even for this very simple example suite, tasks from three or four
different cycle points can in principle run simultaneously at any given
time.

In fact, if our tasks are able to trigger off internal outputs of
upstream tasks (message triggers) rather than waiting on full completion,
then successive instances of the forecast models could overlap as well (because
model restart outputs are generally completed early in the forecast) for an
even more efficient job schedule.

%Finally, we note again that a good job scheduler should be able to
%dynamically adapt to delays in any part of the suite due to resource
%contention, varying run times, or anything else that will inevitably
%modify the depicted job schedules.

\subsection{The Cylc Scheduling Algorithm}
\label{TheCylcSchedulingAlgorithm}

\begin{figure}
    \begin{center}
        \includegraphics[width=8cm]{graphics/png/orig/task-pool.png}
    \end{center}
    \caption[The cylc task pool]{\scriptsize How cylc sees a suite, in
    contrast to the multi-cycle-point dependency graph of
    Figure~\ref{fig-dep-multi}.
    Task colors represent different cycle points, and the small squares
    and circles represent different prerequisites and outputs. A task
    can run when its prerequisites are satisfied by the outputs
    of other tasks in the pool.}
    \label{fig-task-pool}
\end{figure}

Cylc manages a pool of proxy objects that represent the real tasks in a
suite. Task proxies know how to run the real tasks that they represent,
and they receive progress messages from the tasks as they run (usually
reports of completed outputs). There is no global cycling mechanism to
advance the suite; instead individual task proxies have their own
private cycle point and spawn their own successors when the time is
right. Task proxies are self-contained - they know their own
prerequisites and outputs but are not aware of the wider suite.
Inter-cycle dependence is not treated as special, and the task pool can
be populated with tasks with many different cycle points. The task pool
is illustrated in Figure~\ref{fig-task-pool}. {\em Whenever any task
changes state due to completion of an output, every task checks to see
if its own prerequisites have been satisfied.}
%\footnote{In fact this dependency negotiation goes through a broker
%object (rather than every task literally checking every other task)
%which scales as $n$ (rather than $n^2$) where $n$ is the number of task
%proxies in the pool.}
In effect, cylc gets a pool of tasks to self-organize by negotiating
their own dependencies so that optimal scheduling, as described in the
previous section, emerges naturally at run time.

%\pagebreak
\section{Cylc Screenshots}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.8\textwidth]{graphics/png/orig/gcylc-graph-and-dot-views.png}
    \end{center}
\caption[gcylc graph and dot views]{\scriptsize gcylc graph and dot views.}
\label{fig-gcylc-1}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.8\textwidth]{graphics/png/orig/gcylc-text-view.png}
    \end{center}
\caption[gcylc text view]{\scriptsize gcylc text view.}
\label{fig-gcylc-2}
\end{figure}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/gscan.png}
    \end{center}
\caption[gscan multi-suite state summary GUI]{\scriptsize gscan multi-suite state summary GUI.}
\label{fig-gscan}
\end{figure}


\begin{figure}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/ecox-1.png}
    \end{center}
\caption[A large-ish suite graphed by cylc]{\scriptsize A large-ish suite graphed by cylc.}
\label{fig-ecox-1}
\end{figure}

% dump floats
\clearpage

%\pagebreak

\section{Installation}
\label{Requirements}

Cylc runs on Unix variants, usually Linux, and including Apple OS X.

\subsection{External Software Packages}

{\bf Python \lstinline@>=@ 2.6} is required (but not yet Python 3). Python
should already be installed in your Linux system. \url{https://python.org}.

For Cylc's HTTPS communications layer:
\begin{myitemize}
  \item {\bf OpenSSL} - \url{https://www.openssl.org/}
  \item {\bf pyOpenSSL} - \url{http://www.pyopenssl.org/}
  \item {\bf python-requests} - \url{http://docs.python-requests.org/}
  \item ({\bf python-urllib3} - should be bundled with python-requests)
\end{myitemize}

The following packages are highly recommended, but are technically optional as
you can construct and run suites without dependency graph visualisation or
the Cylc GUIs:

\begin{myitemize}
  \item {\bf PyGTK} - GUI toolkit \url{http://www.pygtk.org}.  {\em Note PyGTK
    typically comes with your system Python. It is alledgedly quite
    difficult to install if you need to do so for another Python version.}
    \item {\bf Graphviz} - graph layout engine (tested 2.36.0):
      \url{http://www.graphviz.org}.
    \item {\bf Pygraphviz} - Python Graphviz interface (tested 1.2):
       \url{http://pygraphviz.github.io/}. To build this you may need some {\em
       devel} packages too:
          \begin{myitemize}
              \item python-devel
              \item graphviz-devel
          \end{myitemize}
\end{myitemize}

The User Guide is generated from \LaTeX source files by running
\lstinline=make= in the top level Cylc directory. The specific packages
required may vary by distribution, e.g.:

\begin{myitemize}
    \item texlive
    \item texlive-tocloft
    \item texlive-framed
    \item texlive-preprint (for \lstinline=fullpage.sty=)
    \item texlive-tex4ht
\end{myitemize}

To generate the HTML User Guide {\bf ImageMagick} is also needed.

In most modern Linux distributions all of the software above can be installed
via the system package manager. Otherwise download packages manually and follow
their native installation instructions. To check that all (non \LaTeX packages)
are installed properly:

\lstset{language=transcript}
\begin{lstlisting}
$ cylc check-software
Checking for Python >= 2.6 ... found 2.7.6 ... ok
Checking for non-Python packages:
 + Graphviz ... ok
Checking for Python packages:
 + pygraphviz ... ok
 + pygtk ... ok
\end{lstlisting}

If errors are reported then the packages concerned are either not installed or
not in your Python search path. (Note that \lstinline=cylc check-software= has
become quite trivial as we've removed or bundled some former dependencies, but
in future we intend to make it print a comprehensive list of library versions
etc.\ to include in with bug reports.)

\subsection{Software Bundled With Cylc}

Cylc bundles several third party packages which do not need to be installed
separately.

\begin{myitemize}
  \item {\bf cherrypy 6.0.2} (slightly modified): a pure Python HTTP framework
    that we use as a web server for communication between server processes
    (suite server programs) and client programs (running tasks, GUIs, CLI commands).
    Client communication is via the Python {\bf requests} library if available
    (recommended) or else pure Python via {\bf urllib2}.
\newline \url{http://www.cherrypy.org/}
\newline \url{http://docs.python-requests.org/}
  \item {\bf Jinja2 2.10}: a full featured template engine for Python, and its
    dependency {\bf MarkupSafe 0.23}; both BSD licensed.
\newline \url{http://jinja.pocoo.org/}
\newline \url{http://www.pocoo.org/projects/markupsafe/}
  \item the {\bf xdot} graph viewer (modified), LGPL licensed:
    \newline \url{https://github.com/jrfonseca/xdot.py}
\end{myitemize}

\subsection{Installing Cylc}
\label{InstallCylc}

Cylc releases can be downloaded from from \url{https://cylc.github.io/cylc}.

The wrapper script \lstinline=admin/cylc-wrapper= should be installed as
\lstinline=cylc= in the system executable search path (e.g.\
\lstinline=/usr/local/bin/=) and modified slightly to point to a location
such as \lstinline=/opt= where successive Cylc releases will be unpacked
side by side.

To install Cylc, unpack the release tarball in the right location, e.g.\
\lstinline=/opt/cylc-7.4.0=, type \lstinline=make= inside the release
directory, and set site defaults - if necessary - in a site global config file
(below).

Make a symbolic link from \lstinline=cylc= to the latest installed version:
\lstinline=ln -s /opt/cylc-7.4.0 /opt/cylc=. This will be invoked by the
central wrapper if a specific version is not requested. Otherwise, the
wrapper will attempt to invoke the Cylc version specified in
\lstinline@$CYLC_VERSION@, e.g.\ \lstinline@CYLC_VERSION=7.4.0@. This variable
is automatically set in task job scripts to ensure that jobs use the same Cylc
version as their parent suite server program.  It can also be set by users,
manually or in login scripts, to fix the Cylc version in their environment.

Installing subsequent releases is just a matter of unpacking the new tarballs
next to the previous releases, running \lstinline=make= in them, and copying
in (possibly with modifications) the previous site global config file.

\subsubsection{Local User Installation}
\label{LocalInstall}

It is easy to install Cylc under your own user account if you don't have
root or sudo access to the system: just put the central Cylc wrapper in
\lstinline=$HOME/bin/= (making sure that is in your \lstinline=$PATH=) and
modify it to point to a directory such as \lstinline=$HOME/cylc/= where you
will unpack and install release tarballs. Local installation of third party
dependencies like Graphviz is also possible, but that depends on the particular
installation methods used and is outside of the scope of this document.

\subsubsection{Create A Site Config File}

Site and user global config files define some important parameters that affect
all suites, some of which may need to be customized for your site.
See~\ref{SiteAndUserConfiguration} for how to generate an initial site file and
where to install it. All legal site and user global config items are defined
in~\ref{SiteRCReference}.

\subsubsection{Configure Site Environment on Job Hosts}
\label{Configure Site Environment on Job Hosts}

If your users submit task jobs to hosts other than the hosts they use to run
their suites, you should ensure that the job hosts have the correct environment
for running cylc. A cylc suite generates task job scripts that normally invoke
\lstinline=bash -l=, i.e. it will invoke bash as a login shell to run the job
script. Users and sites should ensure that their bash login profiles are able
to set up the correct environment for running cylc and their task jobs.

Your site administrator may customise the environment for all task jobs by adding
a \lstinline=${CYLC_DIR}/conf/job-init-env.sh= file and populate it with the
appropriate contents. If customisation is still required, you can add your own
\lstinline=${HOME}/.cylc/job-init-env.sh= file and populate it with the
appropriate contents.

\begin{myitemize}
\item \lstinline=${HOME}/.cylc/job-init-env.sh=
\item \lstinline=${CYLC_DIR}/conf/job-init-env.sh=
\end{myitemize}

The job will attempt to source the first of these files it finds to set up its
environment.

\subsection{Automated Tests}
\label{RTAST}

The cylc test battery is primarily intended for developers to check that
changes to the source code don't break existing functionality. Note that
some test failures can be expected to result from suites timing out,
even if nothing is wrong, if you run too many tests in parallel. See
\lstinline=cylc test-battery --help=.

\section{Cylc Terminology}

\subsection{Jobs and Tasks}

A {\em job} is a program or script that runs on a computer, and a {\em task} is
a workflow abstraction - a node in the suite dependency graph - that represents
a job.

\subsection{Cycle Points}

A {\em cycle point} is a particular date-time (or integer) point in a sequence
of date-time (or integer) points. Each cylc task has a private cycle point and
can advance independently to subsequent cycle points. It may sometimes be
convenient, however, to refer to the ``current cycle point'' of a suite (or the
previous or next one, etc.) with reference to a particular task, or in the
sense of all tasks instances that ``belong to'' a particular cycle point. But
keep in mind that different tasks may pass through the ``current cycle point''
(etc.) at different times as the suite evolves.

\section{Workflows For Cycling Systems}
\label{Workflows For Cycling Systems}

A model run and associated processing may need to be cycled for the following
reasons:

\begin{myitemize}
    \item In real time forecasting systems, a new forecast may be initiated
        at regular intervals when new real time data comes in.
    \item It may be convenient (or necessary, e.g.\ due to batch scheduler
        queue limits) to split single long model runs into many smaller chunks,
        each with associated pre- and post-processing workflows.
\end{myitemize}

Cylc provides two ways of constructing workflows for cycling systems: {\em cycling workflows} and {\em parameterized tasks}.

\subsection{Cycling Workflows}
\label{Cycling Workflows}

This is cylc's classic cycling mode as described in the Introduction. Each
instance of a cycling job is represented by a new instance of {\em the same
task}, with a new cycle point. The suite configuration defines patterns for
extending the workflow on the fly, so it can keep running indefinitely if
necessary. For example, to cycle \lstinline=model.exe= on a monthly sequence we
could define a single task \lstinline=model=, an initial cycle point, and a
monthly sequence. Cylc then generates the date-time sequence and creates a new
task instance for each cycle point as it comes up. Workflow dependencies are
defined generically with respect to the ``current cycle point'' of the tasks
involved.

This is the only sensible way to run very large suites or operational suites
that need to continue cycling indefinitely. The cycling is configured with
standards-based ISO 8601 date-time {\em recurrence expressions}. Multiple
cycling sequences can be used at once in the same suite. See
Section~\ref{ConfiguringScheduling}.

\subsection{Parameterized Tasks as a Proxy for Cycling}
\label{Parameterized Tasks as a Proxy for Cycling}

It is also possible to run cycling jobs with a pre-defined static workflow in
which each instance of a cycling job is represented by {\em a different task}:
as far as the abstract workflow is concerned there is no cycling. The sequence
of tasks can be constructed efficiently, however, using cylc's built-in suite
parameters (\ref{Parameterized Cycling}) or explicit Jinja2 loops
(\ref{Jinja2}).

For example, to run \lstinline=model.exe= 12 times on a monthly cycle we could
loop over an integer parameter \lstinline@R = 0, 1, 2, ..., 11@ to define tasks
\lstinline=model-R0, model-R1, model-R2, ...model-R11=, and the parameter
values could be multiplied by the interval \lstinline=P1M= (one month) to get
the start point point for the corresponding model run.

This method is only good for smaller workflows of finite duration because every
single task has to be mapped out in advance, and cylc has to be aware of all of
them throughout the entire run. Additionally Cylc's {\em cycling workflow}
capabilities (above) are more powerful, more flexible, and generally easier to
use (Cylc will generate the cycle point date-times for you, for instance), so
that is the recommended way to drive most cycling systems.

The primary use for parameterized tasks in cylc is to generate ensembles and
other groups of related tasks at the same cycle point, not as a proxy for
cycling.

\subsection{Mixed Cycling Workflows}

For completeness we note that parameterized cycling can be used within a
cycling workflow. For example, in a daily cycling workflow long (daily)
model runs could be split into four shorter runs by parameterized cycling.
A simpler six-hourly cycling workflow should be considered first, however.

\section{Global (Site, User) Configuration Files}
\label{SiteAndUserConfiguration}

Cylc site and user global configuration files contain settings that affect all
suites. Some of these, such as the range of network ports used by cylc,
should be set at site level,
\lstset{language=transcript}
\begin{lstlisting}
# cylc site global config file
/path/to/cylc/conf/global.rc
# Deprecated path to cylc site global config file
/path/to/cylc/conf/siterc/site.rc
\end{lstlisting}
Others, such as the preferred text editor for suite definitions,
can be overridden by users,
\lstset{language=transcript}
\begin{lstlisting}
# cylc user global config file
~/.cylc/global.rc
# Deprecated cylc user global config file
~/.cylc/user.rc
\end{lstlisting}

The \lstinline=cylc get-site-config= command retrieves current
global settings consisting of cylc defaults overridden by site settings,
if any, overridden by user settings, if any.
If you need to generate an example user global config file filled with the
default values commented out (useful as a quick reference), you can do:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc get-site-config | sed 's/^/#/' > ~/.cylc/global.rc.example
\end{lstlisting}
Settings that do not need to be changed should be deleted to reduce conflicts
with site changes and changes in future versions of cylc.
Copy the example file into \lstinline=~/.cylc/global.rc= if you need to apply
any changes.

Legal items, values, and system defaults are documented in
(\ref{SiteRCReference}).

%\pagebreak
\section{Tutorial}
\label{Tutorial}

This section provides a hands-on tutorial introduction to basic cylc
functionality.

\subsection{User Config File}

Some settings affecting cylc's behaviour can be defined in site and user
{\em global config files}. For example, to choose the text editor invoked by
cylc on suite definitions:

\lstset{language=suiterc}
\begin{lstlisting}
# $HOME/.cylc/global.rc
[editors]
    terminal = vim
    gui = gvim -f
\end{lstlisting}

\begin{myitemize}
\item For more on site and user global config files
  see~\ref{SiteAndUserConfiguration} and~\ref{SiteRCReference}.
\end{myitemize}

\subsubsection{Configure Environment on Job Hosts}
\label{Configure Environment on Job Hosts}

See~\ref{Configure Site Environment on Job Hosts} for information.

\subsection{User Interfaces}
\label{CUI}

You should have access to the cylc command line (CLI) and graphical (GUI) user
interfaces once cylc has been installed as described in
Section~\ref{InstallCylc}.

\subsubsection{Command Line Interface (CLI)}

The command line interface is unified under a single top level
\lstinline=cylc= command that provides access to many sub-commands
and their help documentation.

\lstset{language=transcript}
\begin{lstlisting}
$ cylc help       # Top level command help.
$ cylc run --help # Example command-specific help.
\end{lstlisting}

Command help transcripts are printed in~\ref{CommandReference} and are
available from the GUI Help menu.

Cylc is {\em scriptable} - the error status returned by commands can be
relied on.

\subsubsection{Graphical User Interface (GUI)}

The cylc GUI covers the same functionality as the CLI, but it has more
sophisticated suite monitoring capability. It can start and stop suites, or
connect to suites that are already running; in either case, shutting down the
GUI does not affect the suite itself.

\lstset{language=transcript}
\begin{lstlisting}
$ gcylc & # or:
$ cylc gui & # Single suite control GUI.
$ cylc gscan & # Multi-suite monitor GUI.
\end{lstlisting}

Clicking on a suite in gscan, shown in Figure~\ref{fig-gscan}, opens a gcylc
instance for it.

\subsection{Suite Definitions}

Cylc suites are defined by extended-INI format \lstinline=suite.rc=
files (the main file format extension is section nesting). These reside
in {\em suite definition directories} that may also contain a
\lstinline=bin= directory and any other suite-related files.

\begin{myitemize}
\item For more on the suite definition file format, see~\ref{SuiteDefinition}
    and~\ref{SuiteRCReference}.
\end{myitemize}

\subsection{Suite Registration}

Suite registration creates a run directory (under \lstinline=~/cylc-run/= by
default) and populates it with authentication files and a symbolic link to a
suite definition directory. Cylc commands that parse suite definitions can take
the file path or the suite name as input. Commands that interact with running
suites have to target the suite by name.

\lstset{language=transcript}
\begin{lstlisting}
# Target a suite by file path:
$ cylc validate /path/to/my/suite/suite.rc
$ cylc graph /path/to/my/suite/suite.rc

# Register a suite:
$ cylc register my.suite /path/to/my/suite/

# Target a suite by name:
$ cylc graph my.suite
$ cylc validate my.suite
$ cylc run my.suite
$ cylc stop my.suite
# etc.
\end{lstlisting}

\subsection{Suite Passphrases}
\label{tutPassphrases}

Registration (above) also generates a suite-specific passphrase file under
\lstinline=.service/= in the suite run directory. It is loaded by the suite
server program at start-up and used to authenticate connections from client
programs.

Possession of a suite's passphrase file gives full control over it.
Without it, the information avaiable to a client is determined by the suite's
public access privilege level.

For more on connection authentication, suite passphrases, and public access,
see~\ref{ConnectionAuthentication}.


\subsection{Import The Example Suites}
\label{ImportTheExampleSuites}

Run the following command to copy cylc's example suites and register them for
your own use:

\lstset{language=transcript}
\begin{lstlisting}
$ cylc import-examples /tmp
\end{lstlisting}

\subsection{Rename The Imported Tutorial Suites}

Suites can be renames by simply renaming (i.e.\ moving) their run directories.
Make the tutorial suite names shorter, and print their locations with
\lstinline=cylc print=:

\begin{lstlisting}
$ mv ~/cylc-run/$(cylc --version)/examples/tutorial ~/cylc-run/tut
$ cylc print -ya tut
tut/oneoff/jinja2  | /tmp/cylc-examples/7.0.0/tutorial/oneoff/jinja2
tut/cycling/two    | /tmp/cylc-examples/7.0.0/tutorial/cycling/two
tut/cycling/three  | /tmp/cylc-examples/7.0.0/tutorial/cycling/three
# ...
\end{lstlisting}

See \lstinline=cylc print --help= for other display options.

\subsection{Suite Validation}

Suite definitions can be validated to detect syntax (and other) errors:

\lstset{language=transcript}
\begin{lstlisting}
# pass:
$ cylc validate tut/oneoff/basic
Valid for cylc-6.0.0
$ echo $?
0
# fail:
$ cylc validate my/bad/suite
Illegal item: [scheduling]special tusks
$ echo $?
1
\end{lstlisting}

\subsection{Hello World in Cylc}

\hilight{ suite: \lstinline=tut/oneoff/basic= }
\vspace{3mm}

Here's the traditional {\em Hello World} program rendered as a cylc
suite:
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/oneoff/basic/suite.rc}
\lstset{language=transcript}

Cylc suites feature a clean separation of scheduling configuration,
which determines {\em when} tasks are ready to run; and runtime
configuration, which determines {\em what} to run (and {\em where} and
{\em how} to run it) when a task is ready. In this example the
\lstinline=[scheduling]= section defines a single task called
\lstinline=hello= that triggers immediately when the suite starts
up. When the task finishes the suite shuts down. That this is a
{\em dependency graph} will be more obvious when more tasks are added.
Under the \lstinline=[runtime]= section the
\lstinline=script= item defines a simple inlined
implementation for \lstinline=hello=: it sleeps for ten seconds,
then prints \lstinline=Hello World!=, and exits. This ends up in a {\em
job script} generated by cylc to encapsulate the task (below) and,
thanks to some defaults designed to allow quick
prototyping of new suites, it is submitted to run as a background job on
the suite host. In fact cylc even provides a default task implementation
that makes the entire \lstinline=[runtime]= section technically optional:
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/oneoff/minimal/suite.rc}
\lstset{language=transcript}
(the resulting {\em dummy task} just prints out some identifying
information and exits).

\subsection{Editing Suites}

The text editor invoked by cylc on suite definitions is determined
by cylc site and user global config files, as shown above in~\ref{CUI}.
Check that you have renamed the tutorial examples suites as described
just above and open the {\em Hello World} suite definition in your text
editor:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc edit tut/oneoff/basic # in-terminal
$ cylc edit -g tut/oneoff/basic & # or GUI
\end{lstlisting}
Alternatively, start gcylc on the suite:
\lstset{language=transcript}
\begin{lstlisting}
$ gcylc tut/oneoff/basic &
\end{lstlisting}
and choose {\em Suite } \textrightarrow {\em Edit} from the menu.

The editor will be invoked from within the suite definition directory for easy
access to other suite files (in this case there are none). There are syntax
highlighting control files for several text editors under
\lstinline=/path/to/cylc/conf/=; see in-file comments for installation
instructions.

\subsection{Running Suites}
\label{RunningSuitesCLI}

\subsubsection{CLI}
Run \lstinline=tut/oneoff/basic= using the \lstinline=cylc run= command.
As a suite runs detailed timestamped information is written to a {\em suite
log} and progress can be followed with cylc's suite monitoring tools (below).
By default a suite server program daemonizes after printing a short message so
that you can exit the terminal or even log out without killing the suite:

\lstset{language=transcript}
\begin{lstlisting}
$ cylc run tut/oneoff/basic
            ._.
            | |                 The Cylc Suite Engine [7.0.0]
._____._. ._| |_____.           Copyright (C) 2008-2018 NIWA
| .___| | | | | .___|  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| !___| !_! | | !___.  This program comes with ABSOLUTELY NO WARRANTY;
!_____!___. |_!_____!  see `cylc warranty`.  It is free software, you
      .___! |           are welcome to redistribute it under certain
      !_____!                conditions; see `cylc conditions`.

*** listening on nwp-1:43027 ***

To view suite server program contact information:
 $ cylc get-suite-contact tut/oneoff/basic

Other ways to see if the suite is still running:
 $ cylc scan -n '\btut/oneoff/basic\b' nwp-1
 $ cylc ping -v --host=nwp-1 tut/oneoff/basic
 $ ps h -opid,args 123456  # on nwp-1

\end{lstlisting}

If you're quick enough (this example only takes 10-15 seconds to run) the
\lstinline=cylc scan= command will detect the running suite:
\begin{lstlisting}
$ cylc scan
tut/oneoff/basic oliverh@nwp-1:43027
\end{lstlisting}

Note you can use the \lstinline=--no-detach= and \lstinline=--debug= options
to \lstinline=cylc-run= to prevent the suite from daemonizing (i.e.\ to make
it stay attached to your terminal until it exits).

When a task is ready cylc generates a {\em job script} to run it, by
default as a background jobs on the suite host.  The job process ID is
captured, and job output is directed to log files in standard
locations under the suite run directory.

Log file locations relative to the suite run directory look like
\lstinline=job/1/hello/01/= where the first digit is the {\em cycle point} of
the task \lstinline=hello= (for non-cycling tasks this is just `1'); and the
final \lstinline=01= is the {\em submit number} (so that job logs do not get
overwritten if a job is resubmitted for any reason).

The suite shuts down automatically once all tasks have succeeded.

\subsubsection{GUI}

The cylc GUI can start and stop suites, or (re)connect to suites that
are already running:
\begin{lstlisting}
$ cylc gui tut/oneoff/basic &
\end{lstlisting}
Use the tool bar {\em Play} button, or the {\em Control}
\textrightarrow {\em Run} menu item, to run the suite again.
You may want to alter the suite definition slightly to make the task
take longer to run. Try right-clicking on the \lstinline=hello= task
to view its output logs. The relative merits of the three {\em suite
views} - dot, text, and graph - will be more apparent later when we
have more tasks. Closing the GUI does not affect the suite itself.

\subsection{Discovering Running Suites}

Suites that are currently running can be detected with command line or
GUI tools:
\begin{lstlisting}
# list currently running suites and their port numbers:
$ cylc scan
tut/oneoff/basic oliverh@nwp-1:43001

# GUI summary view of running suites:
$ cylc gscan &
\end{lstlisting}

The scan GUI is shown in Figure~\ref{fig-gscan}; clicking on a suite in it
opens gcylc.

\subsection{Task Identifiers}

At run time, task instances are identified by {\em name}, which is
determined entirely by the suite definition, and a {\em cycle point} which is
usually a date-time or an integer:
\lstset{language=transcript}
\begin{lstlisting}
foo.20100808T00Z   # a task with a date-time cycle point
bar.1              # a task with an integer cycle point (could be non-cycling)
\end{lstlisting}
Non-cycling tasks usually just have the cycle point \lstinline=1=, but this
still has to be used to target the task instance with cylc commands.

\subsection{Job Submission: How Tasks Are Executed}

\hilight{ suite: \lstinline=tut/oneoff/jobsub= }
\vspace{3mm}

Task {\em job scripts} are generated by cylc to wrap the task implementation
specified in the suite definition (environment, script, etc.) in
error trapping code, messaging calls to report task progress back to the suite
server program, and so forth. Job scripts are written to the {\em suite job log
directory} where they can be viewed alongside the job output logs. They
can be accessed at run time by right-clicking on the task in the cylc GUI, or
printed to the terminal:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc cat-log tut/oneoff/basic hello.1
\end{lstlisting}
This command can also print the suite log (and stdout and stderr for suites
in daemon mode) and task stdout and stderr logs (see
\lstinline=cylc cat-log --help=).
A new job script can also be generated on the fly for inspection:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc jobscript tut/oneoff/basic hello.1
\end{lstlisting}

Take a look at the job script generated for \lstinline=hello.1= during
the suite run above. The custom scripting should be clearly visible
toward the bottom of the file.

The \lstinline=hello= task in the first tutorial suite defaults to
running as a background job on the suite host. To submit it to the Unix
\lstinline=at= scheduler instead, configure its job submission settings
as in \lstinline=tut/oneoff/jobsub=:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[job]]]
            batch system = at
\end{lstlisting}

Run the suite again after checking that \lstinline=atd= is running on your
system.

Cylc supports a number of different batch systems. Tasks
submitted to external batch queuing systems like \lstinline=at=,
\lstinline=PBS=, \lstinline=SLURM=, \lstinline=Moab=, or
\lstinline=LoadLeveler=, are displayed as {\em submitted} in the cylc GUI until
they start executing.

\begin{myitemize}
\item For more on task job scripts, see~\ref{JobScripts}.
\item For more on batch systems, see~\ref{AvailableMethods}.
\end{myitemize}

\subsection{Locating Suite And Task Output}

If the \lstinline=--no-detach= option is not used, suite stdout and
stderr will be directed to the suite run directory along with the
time-stamped suite log file, and task job scripts and job logs
(task stdout and stderr). The default suite run directory location is
\lstinline=$HOME/cylc-run=:

\lstset{language=transcript}
\begin{lstlisting}
$ tree $HOME/cylc-run/tut/oneoff/basic/
|-- .service              # location of run time service files
|    |-- contact          # detail on how to contact the running suite
|    |-- db               # private suite run database
|    |-- passphrase       # passphrase for client authentication
|    |-- source           # symbolic link to source directory
|    |-- ssl.cert         # SSL certificate for the suite server
|    `-- ssl.pem          # SSL private key
|-- cylc-suite.db         # back compat symlink to public suite run database
|-- share                 # suite share directory (not used in this example)
|-- work                  # task work space (sub-dirs are deleted if not used)
|    `-- 1                   # task cycle point directory (or 1)
|        `-- hello              # task work directory (deleted if not used)
|-- log                   # suite log directory
|   |-- db                   # public suite run database
|   |-- job                  # task job log directory
|   |   `-- 1                   # task cycle point directory (or 1)
|   |       `-- hello              # task name
|   |           |-- 01                # task submission number
|   |           |   |-- job              # task job script
|   |           |   `-- job-activity.log # task job activity log
|   |           |   |-- job.err          # task stderr log
|   |           |   |-- job.out          # task stdout log
|   |           |   `-- job.status       # task status file
|   |           `-- NN -> 01          # symlink to latest submission number
|   `-- suite                # suite server log directory
|       |-- err                 # suite server stderr log (daemon mode only)
|       |-- out                 # suite server stdout log (daemon mode only)
|       `-- log                 # suite server event log (timestamped info)
\end{lstlisting}
The suite run database files, suite environment file,
and task status files are used internally by cylc. Tasks execute in
private \lstinline=work/= directories that are deleted automatically
if empty when the task finishes. The suite
\lstinline=share/= directory is made available to all tasks (by
\lstinline=$CYLC_SUITE_SHARE_DIR=) as a common share space. The task submission
number increments from 1 if a task retries; this is used as a sub-directory of
the log tree to avoid overwriting log files from earlier job submissions.

The top level run directory location can be changed in site and user
config files if necessary, and the suite share and work locations can be
configured separately because of the potentially larger disk space
requirement.

Task job logs can be viewed by right-clicking on tasks in the gcylc
GUI (so long as the task proxy is live in the suite), manually
accessed from the log directory (of course), or printed to the terminal
with the \lstinline=cylc cat-log= command:
\lstset{language=transcript}
\begin{lstlisting}
# suite logs:
$ cylc cat-log    tut/oneoff/basic           # suite event log
$ cylc cat-log -o tut/oneoff/basic           # suite stdout log
$ cylc cat-log -e tut/oneoff/basic           # suite stderr log
# task logs:
$ cylc cat-log    tut/oneoff/basic hello.1   # task job script
$ cylc cat-log -o tut/oneoff/basic hello.1   # task stdout log
$ cylc cat-log -e tut/oneoff/basic hello.1   # task stderr log
\end{lstlisting}
\begin{myitemize}
    \item For a web-based interface to suite and task logs (and much more),
        see {\em Rose} in~\ref{SuiteStorageEtc}.
    \item For more on environment variables supplied to tasks,
    such as \lstinline=$CYLC_SUITE_SHARE_DIR=, see~\ref{TaskExecutionEnvironment}.
\end{myitemize}

\subsection{Remote Tasks}
\label{RemoteTasks}

\hilight{ suite: \lstinline=tut/oneoff/remote= }
\vspace{3mm}

The \lstinline=hello= task in the first two tutorial suites defaults to
running on the suite host. To make it run on a remote host instead
change its runtime configuration as in \lstinline=tut/oneoff/remote=:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[remote]]]
            host = server1.niwa.co.nz
\end{lstlisting}

In general, a {\em task remote} is a user account, other than the account
running the suite server program, where a task job is submitted to run. It can
be on the same machine running the suite or on another machine.

A task remote account must satify several requirements:
\begin{myitemize}

\item Non-interactive ssh must be enabled from the account running the suite
server program to the account for submitting (and managing) the remote task job.

\item Network settings must allow communication {\em back} from the remote task
job to the suite, either by network ports or ssh, unless the last-resort one
way {\em task polling} communication method is used.

\item Cylc must be installed and runnable on the task remote account. Other
software dependencies like graphviz are not required there.

\item Any files needed by a remote task must be installed on the task
host. In this example there is nothing to install because the
implementation of \lstinline=hello= is inlined in the suite definition
and thus ends up entirely contained within the task job script.

\end{myitemize}

If your username is different on the task host, you can add a \lstinline=User=
setting for the relevant host in your \lstinline=~/.ssh/config=.
If you are unable to do so, the \lstinline=[[[remote]]]= section also supports an
\lstinline@owner=username@ item.

If you configure a task account according to the requirements cylc will invoke
itself on the remote account (with a login shell by default) to create log
directories, transfer any essential service files, send the task job script
over, and submit it to run there by the configured batch system.

Remote task job logs are saved to the suite run directory on the task remote,
not on the account running the suite. They can be retrieved by right-clicking
on the task in the GUI, or to have cylc pull them back to the suite account
automatically do this:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[remote]]]
            host = server1.niwa.co.nz
            retrieve job logs = True
\end{lstlisting}

This suite will attempt to \lstinline=rsync= job logs from the remote
host each time a task job completes.

Some batch systems have considerable delays between the time when the job
completes and when it writes the job logs in its normal location. If this is
the case, you can configure an initial delay and retry delays for job log
retrieval by setting some delays. E.g.:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[remote]]]
            host = server1.niwa.co.nz
            retrieve job logs = True
            # Retry after 10 seconds, 1 minute and 3 minutes
            retrieve job logs retry delays = PT10S, PT1M, PT3M
\end{lstlisting}

Finally, if the disk space of the suite host is limited, you may want to set
\lstinline@[[[remote]]]retrieve job logs max size=SIZE@. The value of SIZE can
be anything that is accepted by the \lstinline@--max-size=SIZE@ option of the
\lstinline=rsync= command. E.g.:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
        [[[remote]]]
            host = server1.niwa.co.nz
            retrieve job logs = True
            # Don't get anything bigger than 10MB
            retrieve job logs max size = 10M
\end{lstlisting}

It is worth noting that cylc uses the existence of a job's \lstinline=job.out=
or \lstinline=job.err= in the local file system to indicate a successful job
log retrieval. If \lstinline=retrieve job logs max size=SIZE= is set and both
\lstinline=job.out= and \lstinline=job.err= are bigger than \lstinline=SIZE=
then cylc will consider the retrieval as failed. If retry delays are specified,
this will trigger some useless (but harmless) retries. If this occurs
regularly, you should try the following:

\begin{myitemize}
\item Reduce the verbosity of STDOUT or STDERR from the task.
\item Redirect the verbosity from STDOUT or STDERR to an alternate log file.
\item Adjust the size limit with tolerance to the expected size of STDOUT or STDERR.
\end{myitemize}

\begin{myitemize}
\item For more on remote tasks see~\ref{RunningTasksOnARemoteHost}

\item For more on task communications, see~\ref{TaskComms}.

\item For more on suite passphrases and authentication,
    see~\ref{tutPassphrases} and~\ref{ConnectionAuthentication}.
\end{myitemize}


\subsection{Task Triggering}

\hilight{ suite: \lstinline=tut/oneoff/goodbye= }
\vspace{3mm}

To make a second task called \lstinline=goodbye= trigger after
\lstinline=hello= finishes successfully, return to the original
example, \lstinline=tut/oneoff/basic=, and change the suite graph
as in \lstinline=tut/oneoff/goodbye=:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "hello => goodbye"
\end{lstlisting}
or to trigger it at the same time as \lstinline=hello=,
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "hello & goodbye"
\end{lstlisting}
and configure the new task's behaviour under \lstinline=[runtime]=:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[goodbye]]
        script = "sleep 10; echo Goodbye World!"
\end{lstlisting}

Run \lstinline=tut/oneoff/goodbye= and check the output from the new
task:
\lstset{language=transcript}
\begin{lstlisting}
$ cat ~/cylc-run/tut/oneoff/goodbye/log/job/1/goodbye/01/job.out
  # or
$ cylc cat-log -o tut/oneoff/goodbye goodbye.1
JOB SCRIPT STARTING
cylc (scheduler - 2014-08-14T15:09:30+12): goodbye.1 started at 2014-08-14T15:09:30+12
cylc Suite and Task Identity:
  Suite Name  : tut/oneoff/goodbye
  Suite Host  : oliverh-34403dl.niwa.local
  Suite Port  : 43001
  Suite Owner : oliverh
  Task ID     : goodbye.1
  Task Host   : nwp-1
  Task Owner  : oliverh
  Task Try No.: 1

Goodbye World!
cylc (scheduler - 2014-08-14T15:09:40+12): goodbye.1 succeeded at 2014-08-14T15:09:40+12
JOB SCRIPT EXITING (TASK SUCCEEDED)
\end{lstlisting}

\subsubsection{Task Failure And Suicide Triggering}

\hilight{ suite: \lstinline=tut/oneoff/suicide= }
\vspace{3mm}

Task names in the graph string can be qualified with a state indicator
to trigger off task states other than success:
\lstset{language=suiterc}
\lstset{language=suiterc}
\begin{lstlisting}
    graph = """
 a => b        # trigger b if a succeeds
 c:submit => d # trigger d if c submits
 e:finish => f # trigger f if e succeeds or fails
 g:start  => h # trigger h if g starts executing
 i:fail   => j # trigger j if i fails
            """
\end{lstlisting}

A common use of this is to automate recovery from known modes of failure:
\lstset{language=suiterc}
\begin{lstlisting}
    graph = "goodbye:fail => really_goodbye"
\end{lstlisting}
i.e.\ if task \lstinline=goodbye= fails, trigger another task that
(presumably) really says goodbye.

Failure triggering generally requires use of {\em suicide triggers} as
well, to remove the recovery task if it isn't required (otherwise it
would hang about indefinitely in the waiting state):
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """hello => goodbye
            goodbye:fail => really_goodbye
         goodbye => !really_goodbye # suicide"""
\end{lstlisting}
This means if \lstinline=goodbye= fails, trigger
\lstinline=really_goodbye=; and otherwise, if \lstinline=goodbye=
succeeds, remove \lstinline=really_goodbye= from the suite.

Try running \lstinline=tut/oneoff/suicide=, which also configures
the \lstinline=hello= task's runtime to make it fail, to see how this
works.
\begin{myitemize}
    \item For more on suite dependency graphs see~\ref{ConfiguringScheduling}.
    \item For more on task triggering see~\ref{TriggerTypes}.
\end{myitemize}

\subsection{Runtime Inheritance}

\hilight{ suite: \lstinline=tut/oneoff/inherit= }
\vspace{3mm}

The \lstinline=[runtime]= section is actually a {\em multiple
inheritance} hierarchy. Each subsection is a {\em namespace} that
represents a task, or if it is inherited by other namespaces, a {\em
family}. This allows common configuration to be factored out of related
tasks very efficiently.
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/oneoff/inherit/suite.rc}
The \lstinline=[root]= namespace provides defaults for all tasks in the suite.
Here both tasks inherit \lstinline=script= from \lstinline=root=, which they
customize with different values of the environment variable
\lstinline=$GREETING=. Note that inheritance from \lstinline=root= is
implicit; from other parents an explicit \lstinline@inherit = PARENT@
is required, as shown below.

\begin{myitemize}
\item For more on runtime inheritance, see~\ref{NIORP}.
\end{myitemize}

\subsection{Triggering Families}

\hilight{ suite: \lstinline=tut/oneoff/ftrigger1= }
\vspace{3mm}

Task families defined by runtime inheritance can also be used as
shorthand in graph trigger expressions. To see this, consider two
``greeter'' tasks that trigger off another task \lstinline=foo=:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => greeter_1 & greeter_2"
\end{lstlisting}
If we put the common greeting functionality of \lstinline=greeter_1=
and \lstinline=greeter_2= into a special \lstinline=GREETERS= family,
the graph can be expressed more efficiently like this:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => GREETERS"
\end{lstlisting}
i.e.\ if \lstinline=foo= succeeds, trigger all members of
\lstinline=GREETERS= at once. Here's the full suite with runtime
hierarchy shown:
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/oneoff/ftrigger1/suite.rc}

(Note that we recommend given ALL-CAPS names to task families to help
distinguish them from task names. However, this is just a convention).

Experiment with the \lstinline=tut/oneoff/ftrigger1= suite to see
how this works.

\subsection{Triggering Off Of Families}

\hilight{ suite: \lstinline=tut/oneoff/ftrigger2= }
\vspace{3mm}

Tasks (or families) can also trigger {\em off} other families, but
in this case we need to specify what the trigger means in terms of
the upstream family members. Here's how to trigger another task
\lstinline=bar= if all members of \lstinline=GREETERS= succeed:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """foo => GREETERS
            GREETERS:succeed-all => bar"""
\end{lstlisting}
Verbose validation in this case reports:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc val -v tut/oneoff/ftrigger2
...
Graph line substitutions occurred:
  IN: GREETERS:succeed-all => bar
  OUT: greeter_1:succeed & greeter_2:succeed => bar
...
\end{lstlisting}
Cylc ignores family member qualifiers like \lstinline=succeed-all= on
the right side of a trigger arrow, where they don't make sense, to
allow the two graph lines above to be combined in simple cases:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => GREETERS:succeed-all => bar"
\end{lstlisting}

Any task triggering status qualified by \lstinline=-all= or
\lstinline=-any=, for the members, can be used with a family trigger.
For example, here's how to trigger \lstinline=bar= if all members
of \lstinline=GREETERS= finish (succeed or fail) and any of them them
succeed:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """foo => GREETERS
    GREETERS:finish-all & GREETERS:succeed-any => bar"""
\end{lstlisting}
(use of \lstinline@GREETERS:succeed-any@ by itself here would trigger
\lstinline=bar= as soon as any one member of \lstinline=GREETERS=
completed successfully). Verbose validation now begins to show how
family triggers can simplify complex graphs, even for this tiny
two-member family:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc val -v tut/oneoff/ftrigger2
...
Graph line substitutions occurred:
  IN: GREETERS:finish-all & GREETERS:succeed-any => bar
  OUT: ( greeter_1:succeed | greeter_1:fail ) & \
       ( greeter_2:succeed | greeter_2:fail ) & \
       ( greeter_1:succeed | greeter_2:succeed ) => bar
...
\end{lstlisting}

Experiment with \lstinline=tut/oneoff/ftrigger2= to see how this
works.

\begin{myitemize}
\item For more on family triggering, see~\ref{FamilyTriggers}.
\end{myitemize}

\subsection{Suite Visualization}

\lstset{language=suiterc}
You can style dependency graphs with an optional
\lstinline=[visualization]= section, as shown in
\lstinline=tut/oneoff/ftrigger2=:
\lstset{language=suiterc}
\begin{lstlisting}
[visualization]
    default node attributes = "style=filled"
    [[node attributes]]
        foo = "fillcolor=#6789ab", "color=magenta"
        GREETERS = "fillcolor=#ba9876"
        bar = "fillcolor=#89ab67"
\end{lstlisting}

To display the graph in an interactive viewer:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc graph tut/oneoff/ftrigger2 &    # dependency graph
$ cylc graph -n tut/oneoff/ftrigger2 & # runtime inheritance graph
\end{lstlisting}
It should look like Figure~\ref{fig-tut-hello-multi} (with the
GREETERS family node expanded on the right).
\begin{figure}
    \begin{center}
        \includegraphics[height=0.3\textheight]{graphics/png/orig/tut-hello-multi-1.png}
        \hspace{20mm}
        \includegraphics[height=0.3\textheight]{graphics/png/orig/tut-hello-multi-2.png}
        \hspace{20mm}
        \includegraphics[height=0.3\textheight]{graphics/png/orig/tut-hello-multi-3.png}
    \end{center}
    \caption{The {\em tut/oneoff/ftrigger2} dependency and runtime inheritance graphs}
\label{fig-tut-hello-multi}
\end{figure}

Graph styling can be applied to entire families at once, and custom
``node groups'' can also be defined for non-family groups.


\subsection{External Task Scripts}

\hilight{ suite: \lstinline=tut/oneoff/external= }
\vspace{3mm}

The tasks in our examples so far have all had inlined implementation, in
the suite definition, but real tasks often need to call external
commands, scripts, or executables. To try this, let's return to the
basic Hello World suite and cut the implementation of the task
\lstinline=hello= out to a file \lstinline=hello.sh= in the suite
bin directory:
\lstset{language=bash}
\lstinputlisting{../../../examples/tutorial/oneoff/external/bin/hello.sh}
Make the task script executable, and change the \lstinline=hello= task
runtime section to invoke it:
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/oneoff/external/suite.rc}

If you run the suite now the new greeting from the external task script
should appear in the \lstinline=hello= task stdout log. This works
because cylc automatically adds the suite bin directory to
\lstinline=$PATH= in the environment passed to tasks via their job
scripts. To execute scripts (etc.) located elsewhere you can
refer to the file by its full file path, or set \lstinline=$PATH=
appropriately yourself (this could be done via
\lstinline=$HOME/.profile=, which is sourced at the top of the task job
script, or in the suite definition itself).

Note the use of \lstinline=set -e= above to make the script abort on
error. This allows the error trapping code in the task job script to
automatically detect unforeseen errors.

\subsection{Cycling Tasks}

\hilight{ suite: \lstinline=tut/cycling/one= }
\vspace{3mm}

So far we've considered non-cycling tasks, which finish without spawning
a successor.

Cycling is based around iterating through date-time or integer sequences. A
cycling task may run at each cycle point in a given sequence (cycle). For
example, a sequence might be a set of date-times every 6 hours starting from a
particular date-time. A cycling task may run for each date-time item (cycle
point) in that sequence.

There may be multiple instances of this type of task running in parallel, if
the opportunity arises and their dependencies allow it. Alternatively, a
sequence can be defined with only one valid cycle point - in that case, a task
belonging to that sequence may only run once.

Open the \lstinline=tut/cycling/one= suite:
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/cycling/one/suite.rc}
The difference between cycling and non-cycling suites is all in the
\lstinline=[scheduling]= section, so we will leave the
\lstinline=[runtime]= section alone for now (this will result in
cycling dummy tasks). Note that the graph is now defined under a new
section heading that makes each task under it have a succession of cycle points
ending in $00$ or $12$ hours, between specified initial and final cycle
points (or indefinitely if no final cycle point is given), as shown in
Figure~\ref{fig-tut-one}.

\begin{figure}
    \begin{center}
        %Q Image out of date now
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/tut-one.png}
    \end{center}
    \caption{The \lstinline=tut/cycling/one= suite}
\label{fig-tut-one}
\end{figure}

\lstset{language=transcript}

If you run this suite instances of \lstinline=foo= will spawn in parallel out
to the {\em runahead limit}, and each \lstinline=bar= will trigger off the
corresponding instance of \lstinline=foo= at the same cycle point. The
runahead limit, which defaults to a few cycles but is configurable, prevents
uncontrolled spawning of cycling tasks in suites that are not constrained by
clock triggers in real time operation.

Experiment with \lstinline=tut/cycling/one= to see how cycling tasks work.

\subsubsection{ISO 8601 Date-Time Syntax}

The suite above is a very simple example of a cycling date-time workflow. More
generally, cylc comprehensively supports the ISO 8601 standard for date-time
instants, intervals, and sequences. Cycling graph sections can be specified
using full ISO 8601 recurrence expressions, but these may be simplified
by assuming context information from the suite - namely initial and final cycle
points. One form of the recurrence syntax looks like
\lstinline=Rn/start-date-time/period= (\lstinline=Rn= means run
\lstinline=n= times). In the example above, if the initial cycle point
is always at 00 or 12 hours then \lstinline=[[[T00,T12]]]= could be
written as \lstinline=[[[PT12H]]]=, which is short for
\lstinline=[[[R/initial-cycle-point/PT12H/]]]= - i.e.\ run every 12 hours
indefinitely starting at the initial cycle point. It is possible to add
constraints to the suite to only allow initial cycle points at 00 or 12 hours
e.g.

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 20130808T00
    initial cycle point constraints = T00, T12
\end{lstlisting}
\lstset{language=transcript}

\begin{myitemize}
    %Q Runahead factor now
    \item For a comprehensive description of ISO 8601 based date-time cycling,
        see~\ref{AdvancedCycling}
    \item For more on runahead limiting in cycling suites,
        see~\ref{RunaheadLimit}.
\end{myitemize}

\subsubsection{Inter-Cycle Triggers}
\label{TutInterCyclePointTriggers}

\hilight{ suite: \lstinline=tut/cycling/two= }
\vspace{3mm}

The \lstinline=tut/cycling/two= suite adds inter-cycle dependence
to the previous example:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        # Repeat with cycle points of 00 and 12 hours every day:
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}
For any given cycle point in the sequence defined by the
cycling graph section heading, \lstinline=bar= triggers off
\lstinline=foo= as before, but now \lstinline=foo= triggers off its own
previous instance \lstinline=foo[-PT12H]=. Date-time offsets in
inter-cycle triggers are expressed as ISO 8601 intervals (12 hours
in this case). Figure~\ref{fig-tut-two} shows how this connects the cycling
graph sections together.
\begin{figure}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/tut-two.png}
    \end{center}
    \caption{The \lstinline=tut/cycling/two= suite}
\label{fig-tut-two}
\end{figure}

Experiment with this suite to see how inter-cycle triggers work.
Note that the first instance of \lstinline=foo=, at suite start-up, will
trigger immediately in spite of its inter-cycle trigger, because cylc
ignores dependence on points earlier than the initial cycle point.
However, the presence of an inter-cycle trigger usually implies something
special has to happen at start-up. If a model depends on its own previous
instance for restart files, for example, then some special process has to
generate the initial set of restart files when there is no previous cycle point
to do it. The following section shows one way to handle this in cylc suites.

\subsubsection{Initial Non-Repeating (R1) Tasks}
\label{initial-non-repeating-r1-tasks}
\hilight{ suite: \lstinline=tut/cycling/three= }
\vspace{3mm}

Sometimes we want to be able to run a task at the initial cycle point, but
refrain from running it in subsequent cycles. We can do this by writing an
extra set of dependencies that are only valid at a single date-time cycle
point. If we choose this to be the initial cycle point, these will only apply
at the very start of the suite.

The cylc syntax for writing this single date-time cycle point occurrence is
\lstinline=R1=, which stands for
\lstinline=R1/no-specified-date-time/no-specified-period=.
This is an adaptation of part of the ISO 8601 date-time standard's recurrence
syntax (\lstinline=Rn/date-time/period=) with some special context information
supplied by cylc for the \lstinline=no-specified-*= data.

The \lstinline=1= in the \lstinline=R1= means run once. As we've specified
no date-time, Cylc will use the initial cycle point date-time by default,
which is what we want. We've also missed out specifying the period - this is
set by cylc to a zero amount of time in this case (as it never
repeats, this is not significant).

For example, in \lstinline=tut/cycling/three=:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    cycle point time zone = +13
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep => foo"
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}
\lstset{language=transcript}
This is shown in Figure~\ref{fig-tut-three}.

Note that the time zone has been set to \lstinline=+1300= in this case,
instead of UTC (\lstinline=Z=) as before. If no time zone or UTC mode was
set, the local time zone of your machine will be used in the cycle points.


At the initial cycle point, \lstinline=foo= will depend on
\lstinline=foo[-PT12H]= and also on \lstinline=prep=:
\lstset{language=suiterc}
\begin{lstlisting}
prep.20130808T0000+13 & foo.20130807T1200+13 => foo.20130808T0000+13
\end{lstlisting}
\lstset{language=transcript}

Thereafter, it will just look like e.g.:
\lstset{language=suiterc}
\begin{lstlisting}
foo.20130808T0000+13 => foo.20130808T1200+13
\end{lstlisting}
\lstset{language=transcript}

However, in our initial cycle point example, the dependence on
\lstinline=foo.20130807T1200+13= will be ignored, because that task's cycle
point is earlier than the suite's initial cycle point and so it cannot run.
This means that the initial cycle point dependencies for \lstinline=foo=
actually look like:
\lstset{language=suiterc}
\begin{lstlisting}
prep.20130808T0000+13 => foo.20130808T0000+13
\end{lstlisting}
\lstset{language=transcript}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/tut-three.png}
    \end{center}
    \caption{The \lstinline=tut/cycling/three= suite}
\label{fig-tut-three}
\end{figure}

\begin{myitemize}
    \item \lstinline=R1= tasks can also be used to make something special
        happen at suite shutdown, or at any single cycle point throughout the
        suite run. For a full primer on cycling syntax,
        see~\ref{AdvancedCycling}.
\end{myitemize}


\subsubsection{Integer Cycling}
\label{TutInteger}
\hilight{ suite: \lstinline=tut/cycling/integer= }
\vspace{3mm}

Cylc can do also do integer cycling for repeating workflows that are not
date-time based.

Open the \lstinline=tut/cycling/integer= suite, which is plotted in
Figure~\ref{fig-tut-int}.
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/cycling/integer/suite.rc}

\begin{figure}
    \begin{center}
        \includegraphics[width=0.65\textwidth]{graphics/png/orig/tut-cyc-int.png}
    \end{center}
    \caption{The \lstinline=tut/cycling/integer= suite}
\label{fig-tut-int}
\end{figure}

The integer cycling notation is intended to look similar to the ISO 8601
date-time notation, but it is simpler for obvious reasons. The example suite
illustrates two recurrence forms,
\lstinline=Rn/start-point/period= and
\lstinline=Rn/period/stop-point=, simplified somewhat using suite context
information (namely the initial and final cycle points). The first form is
used to run one special task called \lstinline=start= at start-up, and for the
main cycling body of the suite; and the second form to run another special task
called \lstinline=stop= in the final two cycles. The \lstinline=P= character
denotes period (interval) just like in the date-time notation.
\lstinline=R/1/P2= would generate the sequence of points \lstinline=1,3,5,...=.

\begin{myitemize}
    \item For more on integer cycling, including a more realistic usage example
        see ~\ref{IntegerCycling}.
\end{myitemize}

\subsection{Jinja2}
\hilight{ suite: \lstinline=tut/oneoff/jinja2= }
\vspace{3mm}

Cylc has built in support for the Jinja2 template processor, which
allows us to embed code in suite definitions to generate the
final result seen by cylc.

The \lstinline=tut/oneoff/jinja2= suite illustrates two common
uses of Jinja2: changing suite content or structure based on the value
of a logical switch; and iteratively generating dependencies and runtime
configuration for groups of related tasks:
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/oneoff/jinja2/suite.rc}

To view the result of Jinja2 processing with the Jinja2 flag
\lstinline@MULTI@ set to \lstinline=False=:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc view --jinja2 --stdout tut/oneoff/jinja2
\end{lstlisting}
\lstset{language=suiterc}
\begin{lstlisting}
[meta]
    title = "A Jinja2 Hello World! suite"
[scheduling]
    [[dependencies]]
        graph = "hello"
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
\end{lstlisting}

And with \lstinline=MULTI= set to \lstinline=True=:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc view --jinja2 --stdout tut/oneoff/jinja2
\end{lstlisting}
\lstset{language=suiterc}
\begin{lstlisting}
[meta]
    title = "A Jinja2 Hello World! suite"
[scheduling]
    [[dependencies]]
        graph = "hello => BYE"
[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
    [[BYE]]
        script = "sleep 10; echo Goodbye World!"
    [[ goodbye_0 ]]
        inherit = BYE
    [[ goodbye_1 ]]
        inherit = BYE
    [[ goodbye_2 ]]
        inherit = BYE
\end{lstlisting}

\subsection{Task Retry On Failure}

\hilight{ suite: \lstinline=tut/oneoff/retry= }
\vspace{3mm}

Tasks can be configured to retry a number of times if they fail.
An environment variable \lstinline=$CYLC_TASK_TRY_NUMBER= increments
from $1$ on each successive try, and is passed to the task to allow
different behaviour on the retry:
\lstset{language=suiterc}
\lstinputlisting{../../../examples/tutorial/oneoff/retry/suite.rc}

If a task with configured retries fails, it goes into the {\em retrying} state
until the next retry delay is up, then it resubmits. It only enters the {\em
failed} state on a final definitive failure.

If a task with configured retries is {\em killed} (by \lstinline=cylc kill= or
via the GUI) it goes to the {\em held} state so that the operator can decide
whether to release it and continue the retry sequence or to abort the retry
sequence by manually resetting it to the {\em failed} state.

Experiment with \lstinline=tut/oneoff/retry= to see how this works.

\subsection{Other Users' Suites}

If you have read access to another user's account (even on another host)
it is possible to use \lstinline=cylc monitor= to look at their suite's
progress without full shell access to their account. To do this, you
will need to copy their suite passphrase to
\lstset{language=transcript}
\begin{lstlisting}
    $HOME/.cylc/SUITE_OWNER@SUITE_HOST/SUITE_NAME/passphrase
\end{lstlisting}
(use of the host and owner names is optional here - see~\ref{passphrases})
{\em and} also retrieve the port number of the running suite from:
\begin{lstlisting}
    ~SUITE_OWNER/cylc-run/SUITE_NAME/.service/contact
\end{lstlisting}
Once you have this information, you can run
\begin{lstlisting}
$ cylc monitor --user=SUITE_OWNER --port=SUITE_PORT SUITE_NAME
\end{lstlisting}
to view the progress of their suite.

Other suite-connecting commands work in the same way; see~\ref{RemoteControl}.

\subsection{Other Things To Try}

Almost every feature of cylc can be tested quickly and easily with a
simple dummy suite. You can write your own, or start from one of the
example suites in \lstinline=/path/to/cylc/examples= (see use of
\lstinline=cylc import-examples= above) - they all run ``out the box''
and can be copied and modified at will.

\begin{myitemize}

\item Change the suite runahead limit in a cycling suite.

\item Stop a suite mid-run with \lstinline=cylc stop=, and restart
it again with \lstinline=cylc restart=.

\item Hold (pause) a suite mid-run with \lstinline=cylc hold=,
    then modify the suite definition and \lstinline=cylc reload= it
    before using \lstinline=cylc release= to continue (you can also
    reload without holding).

\item Use the gcylc View menu to show the task state color key and
watch tasks in the \lstinline=task-states= example evolve
as the suite runs.

\item Manually re-run a task that has already completed or failed,
    with \lstinline=cylc trigger=.

\item Use an {\em internal queue} to prevent more than an alotted number
    of tasks from running at once even though they are ready -
   see~\ref{InternalQueues}.

\item Configure task event hooks to send an email, or shut the suite down,
    on task failure.

\end{myitemize}


\section{Suite Name Registration}
\label{SuiteRegistration}

Cylc commands target suites via their names, which are relative path names under
the suite run directory (\lstinline=~/cylc-run/= by default). Suites can be
grouped together under sub-directories.
E.g.:
\begin{lstlisting}
$ cylc print -t nwp
nwp
 |-oper
 | |-region1  Local Model Region1       /home/oliverh/cylc-run/nwp/oper/region1
 | `-region2  Local Model Region2       /home/oliverh/cylc-run/nwp/oper/region2
 `-test
   `-region1  Local Model TEST Region1  /home/oliverh/cylc-run/nwp/test/region1
\end{lstlisting}

Suites can be pre-registered with a name using the \lstinline=cylc register=
command. The creates the essential directory structure for the suite, and
generates some service files underneath it. Otherwise, \lstinline=cylc run= will
create these files on suite start up.

%\pagebreak
\section{Suite Definition}
\label{SuiteDefinition}

Cylc suites are defined in structured, validated, {\em suite.rc} files
that concisely specify the properties of, and the relationships
between, the various tasks managed by the suite. This section of the
User Guide deals with the format and content of the suite.rc file,
including task definition. Task implementation - what's required of the
real commands, scripts, or programs that do the processing that the
tasks represent - is covered in~\ref{TaskImplementation}; and
task job submission - how tasks are submitted to run - is
in~\ref{TaskJobSubmission}.

\subsection{Suite Definition Directories}
\label{SuiteDefinitionDirectories}

A cylc {\em suite definition directory} contains:
\begin{myitemize}
    \item {\bf A suite.rc file}: this is the suite definition.
        \begin{myitemize}
            \item And any include-files used in it (see below; may be
                kept in sub-directories).
        \end{myitemize}
    \item {\bf A \lstinline=bin/= sub-directory} (optional)
        \begin{myitemize}
            \item For scripts and executables that implement, or are
                used by, suite tasks.
            \item Automatically added to \lstinline=$PATH= in task
                execution environments.
            \item Alternatively, tasks can call external
                commands, scripts, or programs; or they can be scripted
                entirely within the suite.rc file.
        \end{myitemize}
    \item {\bf A \lstinline=lib/python/= sub-directory} (optional)
        \begin{myitemize}
            \item For custom job submission modules
                (see~\ref{CustomJobSubmissionMethods})
                and local Python modules imported by custom Jinja2 filters
                (see~\ref{CustomJinja2Filters}).
        \end{myitemize}
    \item {\bf Any other sub-directories and files} - documentation,
        control files, etc. (optional)
        \begin{myitemize}
            \item Holding everything in one place makes proper suite
                revision control possible.
            \item Portable access to files here, for running tasks, is
                provided through
                \lstinline=$CYLC_SUITE_DEF_PATH=
                (see~\ref{TaskExecutionEnvironment}).
            \item Ignored by cylc, but the entire suite definition
                directory tree is copied when you copy a
                suite using cylc commands.

        \end{myitemize}
\end{myitemize}
A typical example:
\lstset{language=transcript}
\begin{lstlisting}
/path/to/my/suite   # suite definition directory
    suite.rc           # THE SUITE DEFINITION FILE
    bin/               # scripts and executables used by tasks
        foo.sh
        bar.sh
        ...
    # (OPTIONAL) any other suite-related files, for example:
    inc/               # suite.rc include-files
        nwp-tasks.rc
        globals.rc
        ...
    doc/               # documentation
    control/           # control files
    ancil/             # ancillary files
    ...
\end{lstlisting}

\subsection{Suite.rc File Overview}
\label{SuiteRCFile}

Suite.rc files are an extended-INI format with section nesting.

Embedded template processor expressions may also be used in the file, to
programatically generate the final suite definition seen by
cylc. Currently the Jinja2 template processor is supported
(\url{http://jinja.pocoo.org/docs}); see~\ref{Jinja2} for examples. In the
future cylc may provide a plug-in interface to allow use of other template
engines too.

\subsubsection{Syntax}
\label{Syntax}

The following defines legal suite.rc syntax:
\begin{myitemize}
    \item {\bf Items} are of the form \lstinline@item = value@.
    \item {\bf [Section]} headings are enclosed in square brackets.
    \item {\bf Sub-section [[nesting]]} is defined by repeated square brackets.
    \item Sections are {\bf closed} by the next section heading.
    \item {\bf Comments} (line and trailing) follow a hash character: \#
    \item {\bf List values} are comma-separated.
    \item {\bf Single-line string values} can be single-, double-, or un-quoted.
    \item {\bf Multi-line string values} are triple-quoted (using
        single or double quote characters).
    \item {\bf Boolean values} are capitalized: True, False.
    \item {\bf Leading and trailing whitespace} is ignored.
    \item {\bf Indentation} is optional but should be used for clarity.
    \item {\bf Continuation lines} follow a trailing backslash: \textbackslash
    \item {\bf Duplicate sections} add their items to those previously
        defined under the same section.
    \item {\bf Duplicate items} override, {\em except for dependency
        \lstinline=graph= strings, which are additive}.
    \item {\bf Include-files} \lstinline=%include inc/foo.rc= can be
        used as a verbatim inlining mechanism.
\end{myitemize}
Suites that embed Jinja2 code (see~\ref{Jinja2}) must
process to raw suite.rc syntax.

\subsubsection{Include-Files}

Cylc has native support for suite.rc include-files, which may help to
organize large suites. Inclusion boundaries are completely arbitrary -
you can think of include-files as chunks of the suite.rc file simply
cut-and-pasted into another file. Include-files may be included
multiple times in the same file, and even nested. Include-file paths
can be specified portably relative to the suite definition directory,
e.g.:
\lstset{language=suiterc}
\begin{lstlisting}
# include the file $CYLC_SUITE_DEF_PATH/inc/foo.rc:
%include inc/foo.rc
\end{lstlisting}

\paragraph{Editing Temporarily Inlined Suites}

Cylc's native file inclusion mechanism supports optional inlined
editing:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc edit --inline SUITE
\end{lstlisting}
The suite will be split back into its constituent include-files when you
exit the edit session. While editing, the inlined file becomes the
official suite definition so that changes take effect whenever you save
the file. See \lstinline=cylc prep edit --help= for more information.

\paragraph{Include-Files via Jinja2}

Jinja2 (\ref{Jinja2}) also has template inclusion functionality.

\subsubsection{Syntax Highlighting For Suite Definitions}
\label{SyntaxHighlighting}

\lstset{language=transcript}
Cylc comes with syntax files for a number of text editors:
\lstset{language=transcript}
\begin{lstlisting}
$CYLC_DIR/conf/cylc.vim     # vim
$CYLC_DIR/conf/cylc-mode.el # emacs
$CYLC_DIR/conf/cylc.lang    # gedit (and other gtksourceview programs)
$CYLC_DIR/conf/cylc.xml     # kate
\end{lstlisting}
Refer to comments at the top of each file to see how to use them.

\subsubsection{Gross File Structure}

Cylc suite.rc files consist of a suite title and description followed by
configuration items grouped under several top level section headings:

\begin{myitemize}
    \item {\bf [cylc] } - {\em non task-specific suite configuration}
    \item {\bf [scheduling] } - {\em determines when tasks are ready to run}
        \begin{myitemize}
            \item tasks with special behaviour, e.g.\ clock-trigger tasks
            \item the dependency graph, which defines the relationships
                between tasks
        \end{myitemize}
    \item {\bf [runtime] } - {\em determines how, where, and what to
        execute when tasks are ready}
        \begin{myitemize}
            \item script, environment, job submission, remote
                hosting, etc.
            \item suite-wide defaults in the {\em root} namespace
            \item a nested family hierarchy with common properties
                inherited by related tasks
        \end{myitemize}
    \item {\bf [visualization] } - suite graph styling
\end{myitemize}


\subsubsection{Validation}
\label{Validation}

Cylc suite.rc files are automatically validated against a specification
that defines all legal entries, values, options, and defaults. This
detects formatting errors, typographic errors, illegal items and illegal
values prior to run time. Some values are complex strings that require
further parsing by cylc to determine their correctness (this is also
done during validation). All legal entries are documented in the {\em
Suite.rc Reference} (\ref{SuiteRCReference}).

The validator reports the line numbers of detected errors. Here's an
example showing a section heading with a missing right bracket:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc validate my.suite
    [[special tasks]
'Section bracket mismatch, line 19'
\end{lstlisting}

If the suite.rc file uses include-files \lstinline=cylc view= will
show an inlined copy of the suite with correct line numbers
(you can also edit suites in a temporarily inlined state with
\lstinline=cylc edit --inline=).

Validation does not check the validity of chosen batch systems.
%this is to allow users to extend cylc with their own job submission
%methods, which are by definition unknown to the suite.rc spec.

\subsection{Scheduling - Dependency Graphs}
\label{ConfiguringScheduling}

\lstset{language=suiterc}
The \lstinline=[scheduling]= section of a suite.rc file defines the
relationships between tasks in a suite - the information that allows
cylc to determine when tasks are ready to run. The most important
component of this is the suite dependency graph. Cylc graph notation
makes clear textual graph representations that are very concise because
sections of the graph that repeat at different hours of the day, say,
only have to be defined once. Here's an example with dependencies that
vary depending on the particular cycle point:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 20200401
    final cycle point = 20200405
    [[dependencies]]
        [[[T00,T06,T12,T18]]] # validity (hours)
            graph = """
A => B & C   # B and C trigger off A
A[-PT6H] => A  # Model A restart trigger
                    """
        [[[T06,T18]]] # hours
            graph = "C => X"
\end{lstlisting}
\lstset{language=transcript}
Figure~\ref{fig-dep-eg-1} shows the complete suite.rc listing alongside
the suite graph.
This is a complete, valid, runnable suite (it will use default
task runtime properties such as \lstinline=script=).

\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
[meta]
    title = "Dependency Example 1"
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20200401
    final cycle point = 20200405
    [[dependencies]]
        [[[T00,T06,T12,T18]]] # validity (hours)
            graph = """
A => B & C   # B and C trigger off A
A[-PT6H] => A  # Model A restart trigger
                    """
        [[[T06,T18]]] # hours
            graph = "C => X"
[visualization]
    initial cycle point = 20200401
    final cycle point = 20200401T06
    [[node attributes]]
        X = "color=red"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/dep-eg-1.png}
    \end{center}
\end{minipage}
\caption[Example Suite]{\scriptsize Example Suite}
\label{fig-dep-eg-1}
\end{figure}

\subsubsection{Graph String Syntax}

Multiline graph strings may contain:
\begin{myitemize}
    \item {\bf blank lines}
    \item {\bf arbitrary white space}
    \item {\bf internal comments:} following the \lstinline=#= character
    \item {\bf conditional task trigger expressions} - see below.
\end{myitemize}

\subsubsection{Interpreting Graph Strings}

Suite dependency graphs can be broken down into pairs in which the left
side (which may be a single task or family, or several that are
conditionally related) defines a trigger for the task or family on the
right. For instance the ``word graph'' {\em C triggers off B which
triggers off A} can be deconstructed into pairs {\em C triggers off B}
and {\em B triggers off A}. In this section we use only the default
trigger type, which is to trigger off the upstream task succeeding;
see~\ref{TriggerTypes} for other available triggers.

In the case of cycling tasks, the triggers defined by a graph string are
valid for cycle points matching the list of hours specified for the
graph section. For example this graph:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "A => B"
\end{lstlisting}
\lstset{language=transcript}
implies that B triggers off A for cycle points in which the hour matches $00$
or $12$.

To define inter-cycle dependencies, attach an offset indicator to the
left side of a pair:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "A[-PT12H] => B"
\end{lstlisting}
\lstset{language=transcript}
This means B[time] triggers off A[time-PT12H] (12 hours before) for cycle
points with hours matching $00$ or $12$. $time$ is implicit because this keeps
graphs clean and concise, given that the majority of tasks will typically
depend only on others with the same cycle point. Cycle point offsets can only
appear on the left of a pair, because a pairs define triggers for the right
task at cycle point $time$. However, \lstinline@A => B[-PT6H]@, which is
illegal, can be reformulated as a {\em future trigger}
\lstinline@A[+PT6H] => B@ (see~\ref{InterCyclePointTriggers}). It is also
possible to combine multiple offsets within a cycle point offset e.g.
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "A[-P1D-PT12H] => B"
\end{lstlisting}
\lstset{language=transcript}
This means that B[Time] triggers off A[time-P1D-PT12H] (1 day and 12 hours
before).

Triggers can be chained together. This graph:
\lstset{language=suiterc}
\begin{lstlisting}
    graph = """A => B  # B triggers off A
               B => C  # C triggers off B"""
\end{lstlisting}
is equivalent to this:
\begin{lstlisting}
    graph = "A => B => C"
\end{lstlisting}
\lstset{language=transcript}

{\em Each trigger in the graph must be unique} but {\em the same task
can appear in multiple pairs or chains}. Separately defined triggers
for the same task have an AND relationship. So this:
\lstset{language=suiterc}
\begin{lstlisting}
    graph = """A => X  # X triggers off A
               B => X  # X also triggers off B"""
\end{lstlisting}

is equivalent to this:
\lstset{language=suiterc}
\begin{lstlisting}
    graph = "A & B => X"  # X triggers off A AND B
\end{lstlisting}
\lstset{language=transcript}

In summary, the branching tree structure of a dependency graph can
be partitioned into lines (in the suite.rc graph string) of pairs
or chains, in any way you like, with liberal use of internal white space
and comments to make the graph structure as clear as possible.

\begin{lstlisting}
# B triggers if A succeeds, then C and D trigger if B succeeds:
    graph = "A => B => C & D"
# which is equivalent to this:
    graph = """A => B => C
               B => D"""
# and to this:
    graph = """A => B => D
               B => C"""
# and to this:
    graph = """A => B
               B => C
               B => D"""
# and it can even be written like this:
    graph = """A => B # blank line follows:

               B => C # comment ...
               B => D"""
\end{lstlisting}

\paragraph{Splitting Up Long Graph Lines}

\lstset{language=suiterc}

It is not necessary to use the general line continuation marker
\lstinline=\= to split long graph lines. Just break at dependency arrows,
or split long chains into smaller ones. This graph:
\begin{lstlisting}
    graph = "A => B => C"
\end{lstlisting}

is equivalent to this:
\begin{lstlisting}
    graph = """A => B =>
                 C"""
\end{lstlisting}

and also to this:
\begin{lstlisting}
    graph = """A => B
               B => C"""
\end{lstlisting}

\subsubsection{Graph Types}
\label{GraphTypes}

A suite definition can contain multiple graph strings that are combined
to generate the final graph.

\paragraph{One-off (Non-Cycling)}

Figure~\ref{fig-test1} shows a small suite of one-off non-cycling
tasks; these all share a single cycle point (\lstinline=1=) and don't spawn
successors (once they're all finished the suite just exits). The integer
\lstinline=1= attached to each graph node is just an arbitrary label here.
\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
[meta]
    title = some one-off tasks
[scheduling]
    [[dependencies]]
        graph = "foo => bar & baz => qux"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=0.25\textwidth]{graphics/png/orig/test1.png}
    \end{center}
\end{minipage}
\caption[One-off (Non-Cycling) Tasks]{\scriptsize One-off (Non-Cycling) Tasks.}
\label{fig-test1}
\end{figure}

\paragraph{Cycling Graphs}

For cycling tasks the graph section heading defines a sequence of cycle points
for which the subsequent graph section is valid. Figure~\ref{fig-test2} shows
a small suite of cycling tasks.
\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
[meta]
    title = some cycling tasks
# (no dependence between cycle points)
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "foo => bar & baz => qux"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/test2.png}
    \end{center}
\end{minipage}
\caption[Cycling Tasks]{\scriptsize Cycling Tasks.}
\label{fig-test2}
\end{figure}

\subsubsection{Graph Section Headings}

Graph section headings define recurrence expressions, the graph within a graph
section heading defines a workflow at each point of the recurrence. For
example in the following scenario:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[ T06 ]]]  # A graph section heading
            graph = foo => bar
\end{lstlisting}
\lstset{language=transcript}

\lstinline=T06= means "Run every day starting at 06:00 after the
initial cycle point". Cylc allows you to start (or end) at any particular
time, repeat at whatever frequency you like, and even optionally limit the
number of repetitions.

Graph section heading can also be used with integer cycling see
\ref{IntegerCycling}.

\paragraph{Syntax Rules}

Date-time cycling information is made up of a starting {\em date-time}, an
{\em interval}, and an optional {\em limit}.

The time is assumed to be in the local time zone unless you set
\lstinline=[cylc]cycle point time zone= or \lstinline=[cylc]UTC mode=. The
calendar is assumed to be the proleptic Gregorian calendar unless you set
\lstinline=[scheduling]cycling mode=.

The syntax for representations is based on the ISO 8601 date-time standard.
This includes the representation of {\em date-time}, {\em interval}. What we
define for cylc's cycling syntax is our own optionally-heavily-condensed form
of ISO 8601 recurrence syntax. The most common full form is:
\lstinline=R[limit?]/[date-time]/[interval]=. However, we allow omitting
information that can be guessed from the context (rules below). This means
that it can be written as:
\begin{lstlisting}
R[limit?]/[date-time]
R[limit?]//[interval]
[date-time]/[interval]
R[limit?] # Special limit of 1 case
[date-time]
[interval]
\end{lstlisting}

with example graph headings for each form being:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R5/T00 ]]]           # Run 5 times at 00:00 every day
[[[ R//PT1H ]]]          # Run every hour (Note the R// is redundant)
[[[ 20000101T00Z/P1D ]]] # Run every day starting at 00:00 1st Jan 2000
[[[ R1 ]]]               # Run once at the initial cycle point
[[[ 20000101T00Z ]]]     # Run once at 00:00 1st Jan 2000
[[[ P1Y ]]]              # Run every year
\end{lstlisting}

Note that \lstinline=T00= is an example of \lstinline=[date-time]=, with an
inferred 1 day period and no limit.

Where some or all {\em date-time} information is omitted, it is inferred to
be relative to the initial date-time cycle point. For example, \lstinline=T00=
by itself would mean the next occurrence of midnight that follows, or is, the
initial cycle point. Entering \lstinline=+PT6H= would mean 6 hours after the
initial cycle point. Entering \lstinline=-P1D= would mean 1 day before the
initial cycle point. Entering no information for the {\em date-time} implies
the initial cycle point date-time itself.

Where the {\em interval} is omitted and some (but not all) {\em date-time}
information is omitted, it is inferred to be a single unit above
the largest given specific {\em date-time } unit. For example, the largest
given specific unit in \lstinline=T00= is hours, so the inferred interval is
1 day (daily), \lstinline=P1D=.

Where the {\em limit} is omitted, unlimited cycling is assumed. This will be
bounded by the final cycle point's date-time if given.

Another supported form of ISO 8601 recurrence is:
\lstinline=R[limit?]/[interval]/[date-time]=. This form uses the
{\em date-time } as the end of the cycling sequence rather than the start.
For example, \lstinline=R3/P5D/20140430T06= means:
\begin{lstlisting}
20140420T06
20140425T06
20140430T06
\end{lstlisting}

This kind of form can be used for specifying special behaviour near the end of
the suite, at the final cycle point's date-time. We can also represent this in
cylc with a collapsed form:
\begin{lstlisting}
R[limit?]/[interval]
R[limit?]//[date-time]
[interval]/[date-time]
\end{lstlisting}

So, for example, you can write:
\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1//+P0D ]]]  # Run once at the final cycle point
[[[ R5/P1D ]]]    # Run 5 times, every 1 day, ending at the final
                  # cycle point
[[[ P2W/T00 ]]]   # Run every 2 weeks ending at 00:00 following
                  # the final cycle point
[[[ R//T00 ]]]    # Run every 1 day ending at 00:00 following the
                  # final cycle point
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Referencing The Initial And Final Cycle Points}
\label{referencing-the-initial-and-final-cycle-points}

For convenience the caret and dollar symbols may be used as shorthand for the
initial and final cycle points. Using this shorthand you can write:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1/^+PT12H ]]]  # Repeat once 12 hours after the initial cycle point
                    # R[limit]/[date-time]
                    # Equivalent to [[[ R1/+PT12H ]]]
[[[ R1/$ ]]]        # Repeat once at the final cycle point
                    # R[limit]/[date-time]
                    # Equivalent to [[[ R1//+P0D ]]]
[[[ $-P2D/PT3H ]]]  # Repeat 3 hourly starting two days before the
                    # [date-time]/[interval]
                    # final cycle point
\end{lstlisting}
\lstset{language=transcript}

Note that there can be multiple ways to write the same headings, for instance
the following all run once at the final cycle point:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1/P0Y ]]]      # R[limit]/[interval]
[[[ R1/P0Y/$ ]]]    # R[limit]/[interval]/[date-time]
[[[ R1/$ ]]]        # R[limit]/[date-time]
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Excluding Dates}
\label{excluding-dates}
\lstset{language=suiterc}

Date-times can be excluded from a recurrence by an exclamation mark for example
\lstinline=[[[ PT1D!20000101 ]]]= means run daily except on the
first of January 2000.

This syntax can be used to exclude one or multiple date-times from a recurrence.
Multiple date-times are excluded using the syntax
\lstinline=[[[ PT1D!(20000101,20000102,...) ]]]=. All date-times listed within
the parentheses after the exclamation mark will be excluded. Note that the
\lstinline=^= and \lstinline=$= symbols (shorthand for the initial
and final cycle points) are both date-times so \lstinline=[[[ T12!$-PT1D ]]]=
is valid.

If using a run limit in combination with an exclusion, the heading might not
run the number of times specified in the limit. For example in the following
suite \lstinline=foo= will only run once as its second run has been excluded.

\begin{lstlisting}
[scheduling]
    initial cycle point = 20000101T00Z
    final cycle point = 20000105T00Z
    [[dependencies]]
        [[[ R2/P1D!20000102 ]]]
            graph = foo
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Advanced exclusion syntax}

In addition to excluding isolated date-time points or lists of date-time points
from recurrences, exclusions themselves may be date-time recurrence sequences.
Any partial date-time or sequence given after the exclamation mark will be
excluded from the main sequence.

For example, partial date-times can be excluded using the syntax:
\lstset{language=suiterc}
\begin{lstlisting}
[[[ PT1H ! T12 ]]]          # Run hourly but not at 12:00 from the inital
                            # cycle point.
[[[ T-00 ! (T00, T06, T12, T18) ]]]   # Run hourly but not at 00:00, 06:00,
                                      # 12:00, 18:00.
[[[ PT5M ! T-15 ]]]         # Run 5-minutely but not at 15 minutes past the
                            # hour from the initial cycle point.
[[[ T00 ! W-1T00 ]]]        # Run daily at 00:00 except on Mondays.
\end{lstlisting}
\lstset{language=transcript}

It is also valid to use sequences for exclusions. For example:
\lstset{language=suiterc}
\begin{lstlisting}
[[[ PT1H ! PT6H ]]]         # Run hourly from the initial cycle point but
                            # not 6-hourly from the intial cycle point.
[[[ T-00 ! PT6H ]]]         # Run hourly on the hour but not 6-hourly
                            # on the hour.
    # Same as [[[ T-00 ! T-00/PT6H ]]] (T-00 context is implied)
    # Same as [[[ T-00 ! (T00, T06, T12, T18) ]]]
    # Same as [[[ PT1H ! (T00, T06, T12, T18) ]]] Initial cycle point dependent

[[[ T12 ! T12/P15D ]]]      # Run daily at 12:00 except every 15th day.

[[[ R/^/P1H ! R5/20000101T00/P1D ]]]    # Any valid recurrence may be used to
                                        # determine exclusions. This example
                                        # translates to: Repeat every hour from
                                        # the initial cycle point, but exclude
                                        # 00:00 for 5 days from the 1st January
                                        # 2000.

\end{lstlisting}
\lstset{language=transcript}

You can combine exclusion sequences and single point exclusions within a
comma separated list enclosed in parentheses:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ T-00 ! (20000101T07, PT2H) ]]]      # Run hourly on the hour but not at 07:00
                                        # on the 1st Jan, 2000 and not 2-hourly
                                        # on the hour.
\end{lstlisting}
\lstset{language=transcript}


\paragraph{How Multiple Graph Strings Combine}
\label{HowMultipleGraphStringsCombine}

For a cycling graph with multiple validity sections for different
hours of the day, the different sections {\em add} to generate the
complete graph. Different graph sections can overlap (i.e.\ the same
hours may appear in multiple section headings) and the same tasks may
appear in multiple sections, but individual dependencies should be
unique across the entire graph. For example, the following graph defines
a duplicate prerequisite for task C:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            graph = "A => B => C"
        [[[T06,T18]]]
            graph = "B => C => X"
            # duplicate prerequisite: B => C already defined at T06, T18
\end{lstlisting}
\lstset{language=transcript}
This does not affect scheduling, but for the sake of clarity and brevity
the graph should be written like this:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            graph = "A => B => C"
        [[[T06,T18]]]
            # X triggers off C only at 6 and 18 hours
            graph = "C => X"
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Advanced Examples}
\label{AdvancedCycling}

The following examples show the various ways of writing graph headings in cylc.
\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1 ]]]         # Run once at the initial cycle point
[[[ P1D ]]]        # Run every day starting at the initial cycle point
[[[ PT5M ]]]       # Run every 5 minutes starting at the initial cycle
                   # point
[[[ T00/P2W ]]]    # Run every 2 weeks starting at 00:00 after the
                   # initial cycle point
[[[ +P5D/P1M ]]]   # Run every month, starting 5 days after the initial
                   # cycle point
[[[ R1/T06 ]]]     # Run once at 06:00 after the initial cycle point
[[[ R1/P0Y ]]]     # Run once at the final cycle point
[[[ R1/$ ]]]       # Run once at the final cycle point (alternative
                   # form)
[[[ R1/$-P3D ]]]   # Run once three days before the final cycle point
[[[ R3/T0830 ]]]   # Run 3 times, every day at 08:30 after the initial
                   # cycle point
[[[ R3/01T00 ]]]   # Run 3 times, every month at 00:00 on the first
                   # of the month after the initial cycle point
[[[ R5/W-1/P1M ]]] # Run 5 times, every month starting on Monday
                   # following the initial cycle point
[[[ T00!^ ]]]      # Run at the first occurrence of T00 that isn't the
                   # initial cycle point
[[[ PT1D!20000101 ]]]  # Run every day days excluding 1st Jan 2000
[[[ 20140201T06/P1D ]]]    # Run every day starting at 20140201T06
[[[ R1/min(T00,T06,T12,T18) ]]]  # Run once at the first instance
                                 # of either T00, T06, T12 or T18
                                 # starting at the initial cycle
                                 # point
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Advanced Starting Up}
\label{AdvancedStartingUp}

Dependencies that are only valid at the initial cycle point can be written
using the \lstinline=R1= notation (e.g.\ as
in~\ref{initial-non-repeating-r1-tasks}. For example:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep => foo"
        [[[T00]]]
            graph = "foo[-P1D] => foo => bar"
\end{lstlisting}
\lstset{language=transcript}

In the example above, \lstinline=R1= implies \lstinline=R1/20130808T00=, so
\lstinline=prep= only runs once at that cycle point (the initial cycle point).
At that cycle point, \lstinline=foo= will have a dependence on
\lstinline=prep= - but not at subsequent cycle points.

However, it is possible to have a suite that has multiple effective initial
cycles - for example, one starting at \lstinline=T00= and another starting
at \lstinline=T12=. What if they need to share an initial task?

Let's suppose that we add the following section to the suite example above:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep => foo"
        [[[T00]]]
            graph = "foo[-P1D] => foo => bar"
        [[[T12]]]
            graph = "baz[-P1D] => baz => qux"
\end{lstlisting}
\lstset{language=transcript}

We'll also say that there should be a starting dependence between
\lstinline=prep= and our new task \lstinline=baz= - but we still want to have
a single \lstinline=prep= task, at a single cycle.

We can write this using a special case of the \lstinline=task[-interval]= syntax -
if the interval is null, this implies the task at the initial cycle point.

For example, we can write our suite like~\ref{fig-test4}.

\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130812T00
    [[dependencies]]
        [[[R1]]]
            graph = "prep"
        [[[R1/T00]]]
# ^ implies the initial cycle point:
     graph = "prep[^] => foo"
        [[[R1/T12]]]
# ^ is initial cycle point, as above:
     graph = "prep[^] => baz"
        [[[T00]]]
     graph = "foo[-P1D] => foo => bar"
        [[[T12]]]
     graph = "baz[-P1D] => baz => qux"
[visualization]
    initial cycle point = 20130808T00
    final cycle point = 20130810T00
    [[node attributes]]
        foo = "color=red"
        bar = "color=orange"
        baz = "color=green"
        qux = "color=blue"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/test4.png}
    \end{center}
\end{minipage}
\caption[Staggered Start Suite]{\scriptsize Staggered Start Suite}
\label{fig-test4}
\end{figure}

This neatly expresses what we want - a task running at the initial cycle point
that has one-off dependencies with other task sets at different cycles.

\begin{figure}
\begin{minipage}[h]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130808T18
    [[dependencies]]
        [[[R1]]]
            graph = "setup_foo => foo"
        [[[+PT6H/PT6H]]]
            graph = """
                foo[-PT6H] => foo
                foo => bar
            """
[visualization]
    initial cycle point = 20130808T00
    final cycle point = 20130808T18
    [[node attributes]]
        foo = "color=red"
        bar = "color=orange"
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[h]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/test5.png}
    \end{center}
\end{minipage}
\caption[Restricted First Cycle Point Suite]{
    \scriptsize Restricted First Cycle Point Suite}
\label{fig-test5}
\end{figure}


A different kind of requirement is displayed in Figure \ref{fig-test5}.
Usually, we want to specify additional tasks and dependencies at the initial
cycle point. What if we want our first cycle point to be entirely special, with
some tasks missing compared to subsequent cycle points?

In Figure \ref{fig-test5}, \lstinline=bar= will not be run at the initial
cycle point, but will still run at subsequent cycle points.
\lstinline=[[[+PT6H/PT6H]]]= means start at \lstinline=+PT6H= (6 hours after
the initial cycle point) and then repeat every \lstinline=PT6H= (6 hours).

Some suites may have staggered start-up sequences where different tasks need
running once but only at specific cycle points, potentially due to differing
data sources at different cycle points with different possible initial cycle
points. To allow this cylc provides a \lstinline=min( )= function that can be
used as follows:

\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20100101T03
    [[dependencies]]
        [[[R1/min(T00,T12)]]]
            graph = "prep1 => foo"
        [[[R1/min(T06,T18)]]]
            graph = "prep2 => foo"
        [[[T00,T06,T12,T18]]]
            graph = "foo => bar"

\end{lstlisting}
\lstset{language=transcript}


In this example the initial cycle point is \lstinline=20100101T03=, so the
\lstinline=prep1= task will run once at \lstinline=20100101T12= and the
\lstinline=prep2= task will run once at \lstinline=20100101T06= as these are
the first cycle points after the initial cycle point in the respective
\lstinline=min( )= entries.


\paragraph{Integer Cycling}
\label{IntegerCycling}

In addition to non-repeating and date-time cycling workflows, cylc can do
integer cycling for repeating workflows that are not date-time based.

To construct an integer cycling suite, set
\lstinline@[scheduling]cycling mode = integer@, and specify integer values for
the initial and (optional) final cycle points. The notation for intervals,
offsets, and recurrences (sequences) is similar to the date-time cycling
notation, except for the simple integer values.

The full integer recurrence expressions supported are:
\begin{myitemize}
    \item \lstinline@Rn/start-point/interval # e.g. R3/1/P2@
    \item \lstinline@Rn/interval/end-point # e.g. R3/P2/9@
\end{myitemize}
But, as for date-time cycling, sequence start and end points can be omitted
where suite initial and final cycle points can be assumed. Some examples:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1 ]]]        # Run once at the initial cycle point
                  # (short for R1/initial-point/?)
[[[ P1 ]]]        # Repeat with step 1 from the initial cycle point
                  # (short for R/initial-point/P1)
[[[ P5 ]]]        # Repeat with step 5 from the initial cycle point
                  # (short for R/initial-point/P5)
[[[ R2//P2 ]]]    # Run twice with step 3 from the initial cycle point
                  # (short for R2/initial-point/P2)
[[[ R/+P1/P2 ]]]  # Repeat with step 2, from 1 after the initial cycle point
[[[ R2/P2 ]]]     # Run twice with step 2, to the final cycle point
                  # (short for R2/P2/final-point)
[[[ R1/P0 ]]]     # Run once at the final cycle point
                  # (short for R1/P0/final-point)
\end{lstlisting}

\subparagraph{Example}

The tutorial illustrates integer cycling in~\ref{TutInteger}, and
\lstinline=$CYLC_DIR/examples/satellite/= is a
self-contained example of a realistic use for integer cycling. It simulates
the processing of incoming satellite data: each new dataset arrives after a
random (as far as the suite is concerned) interval, and is labeled by an
arbitrary (as far as the suite is concerned) ID in the filename. A task called
\lstinline=get_data= at the top of the repeating workflow waits on the next
dataset and, when it finds one, moves it to a cycle-point-specific shared
workspace for processing by the downstream tasks. When \lstinline=get_data.1=
finishes, \lstinline=get_data.2= triggers and begins waiting for the next
dataset at the same time as the downstream tasks in cycle point 1 are
processing the first one, and so on. In this way multiple datasets can be
processed at once if they happen to come in quickly. A single shutdown task
runs at the end of the final cycle to collate results. The suite graph is
shown in Figure~\ref{fig-satellite}.

\begin{figure}
    \begin{center}
        \includegraphics[width=0.4\textwidth]{graphics/png/orig/satellite.png}
    \end{center}
    \caption{The \lstinline=examples/satellite= integer suite}
\label{fig-satellite}
\end{figure}

\subparagraph{Advanced Integer Cycling Syntax}

The same syntax used to reference the initial and final cycle points
(introduced in~\ref{referencing-the-initial-and-final-cycle-points}) for
use with date-time cycling can also be used for integer cycling. For
example you can write:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R1/^ ]]]     # Run once at the initial cycle point
[[[ R1/$ ]]]     # Run once at the final cycle point
[[[ R3/^/P2 ]]]  # Run three times with step two starting at the
                 # initial cycle point
\end{lstlisting}
\lstset{language=transcript}

Likewise the syntax introduced in~\ref{excluding-dates} for excluding
a particular point from a recurrence also works for integer cycling. For
example:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R/P4!8 ]]]       # Run with step 4, to the final cycle point
                     # but not at point 8
[[[ R3/3/P2!5 ]]]    # Run with step 2 from point 3 but not at
                     # point 5
[[[ R/+P1/P6!14 ]]]  # Run with step 6 from 1 step after the
                     # initial cycle point but not at point 14
\end{lstlisting}
\lstset{language=transcript}

Multiple integer exclusions are also valid in the same way as the syntax
in~\ref{excluding-dates}. Integer exclusions may be a list of single
integer points, an integer sequence, or a combination of both:

\lstset{language=suiterc}
\begin{lstlisting}
[[[ R/P1!(2,3,7) ]]]  # Run with step 1 to the final cycle point,
                      # but not at points 2, 3, or 7.
[[[ P1 ! P2 ]]]       # Run with step 1 from the initial to final
                      # cycle point, skipping every other step from
                      # the initial cycle point.
[[[ P1 ! +P1/P2 ]]]   # Run with step 1 from the initial cycle point,
                      # excluding every other step beginning one step
                      # after the initial cycle point.
[[[ P1 !(P2,6,8) ]]]  # Run with step 1 from the intial cycle point,
                      # excluding every other step, and also excluding
                      # steps 6 and 8.
\end{lstlisting}
\lstset{language=transcript}


\subsubsection{Trigger Types}
\label{TriggerTypes}

\lstset{language=suiterc}

Trigger type, indicated by {\em:type} after the upstream task (or family)
name, determines what kind of event results in the downstream task (or
family) triggering.

\paragraph{Success Triggers}

The default, with no trigger type specified, is to trigger off the
upstream task succeeding:
\begin{lstlisting}
# B triggers if A SUCCEEDS:
    graph = "A => B"
\end{lstlisting}
For consistency and completeness, however, the success trigger can be explicit:
\begin{lstlisting}
# B triggers if A SUCCEEDS:
    graph = "A => B"
# or:
    graph = "A:succeed => B"
\end{lstlisting}

\paragraph{Failure Triggers}

To trigger off the upstream task reporting failure:
\begin{lstlisting}
# B triggers if A FAILS:
    graph = "A:fail => B"
\end{lstlisting}
{\em Suicide triggers} can be used to remove task \lstinline=B= here if
\lstinline=A= does not fail, see~\ref{SuicideTriggers}.

\paragraph{Start Triggers}

To trigger off the upstream task starting to execute:
\begin{lstlisting}
# B triggers if A STARTS EXECUTING:
    graph = "A:start => B"
\end{lstlisting}
This can be used to trigger tasks that monitor other tasks once they
(the target tasks) start executing. Consider a long-running forecast model,
for instance, that generates a sequence of output files as it runs. A
postprocessing task could be launched with a start trigger on the model
(\lstinline@model:start => post@) to process the model output as it
becomes available. Note, however, that there are several alternative
ways of handling this scenario: both tasks could be triggered at the
same time (\lstinline@foo => model & post@), but depending on
external queue delays this could result in the monitoring task starting
to execute first; or a different postprocessing task could be
triggered off a message output for each data file
(\lstinline@model:out1 => post1@ etc.; see~\ref{MessageTriggers}), but this
may not be practical if the
number of output files is large or if it is difficult to add cylc
messaging calls to the model.

\paragraph{Finish Triggers}

To trigger off the upstream task succeeding or failing, i.e.\ finishing
one way or the other:
\begin{lstlisting}
# B triggers if A either SUCCEEDS or FAILS:
    graph = "A | A:fail => B"
# or
    graph = "A:finish => B"
\end{lstlisting}

\paragraph{Message Triggers}
\label{MessageTriggers}

Tasks can also trigger off custom output messages. These must be registered in
the \lstinline=[runtime]= section of the emitting task, and reported using the
\lstinline=cylc message= command in task scripting. The graph trigger notation
refers to the item name of the registered output message.
The example suite \lstinline=$CYLC_DIR/examples/message-triggers= illustrates
message triggering.

\lstset{language=suiterc}
\lstinputlisting{../../../examples/message-triggers/suite.rc}

\paragraph{Job Submission Triggers}

It is also possible to trigger off a task submitting, or failing to submit:
\begin{lstlisting}
# B triggers if A submits successfully:
    graph = "A:submit => B"
# D triggers if C fails to submit successfully:
    graph = "C:submit-fail => D"
\end{lstlisting}

A possible use case for submit-fail triggers: if a task goes into the
submit-failed state, possibly after several job submission retries,
another task that inherits the same runtime but sets a different job
submission method and/or host could be triggered to, in effect, run the
same job on a different platform.


\paragraph{Conditional Triggers}

AND operators (\lstinline=&=) can appear on both sides of an arrow. They
provide a concise alternative to defining multiple triggers separately:
\begin{lstlisting}
# 1/ this:
    graph = "A & B => C"
# is equivalent to:
    graph = """A => C
               B => C"""
# 2/ this:
    graph = "A => B & C"
# is equivalent to:
    graph = """A => B
               A => C"""
# 3/ and this:
    graph = "A & B => C & D"
# is equivalent to this:
    graph = """A => C
               B => C
               A => D
               B => D"""
\end{lstlisting}

OR operators (\lstinline=|=) which result in true conditional triggers,
can only appear on the left,\footnote{An OR
operator on the right doesn't make much sense: if ``B or C'' triggers
off A, what exactly should cylc do when A finishes?}
\begin{lstlisting}
# C triggers when either A or B finishes:
    graph = "A | B => C"
\end{lstlisting}

Forecasting suites typically have simple conditional
triggering requirements, but any valid conditional expression can be
used, as shown in Figure~\ref{fig-conditional}
(conditional triggers are plotted with open arrow heads).
\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
        graph = """
# D triggers if A or (B and C) succeed
A | B & C => D
# just to align the two graph sections
D => W
# Z triggers if (W or X) and Y succeed
(W|X) & Y => Z
                """
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/conditional-triggers.png}
    \end{center}
\end{minipage}
\caption[Conditional Triggers] {\scriptsize
Conditional triggers are plotted with open arrow heads.}
\label{fig-conditional}
\end{figure}

\paragraph{Suicide Triggers}
\label{SuicideTriggers}

Suicide triggers take tasks out of the suite. This can be used for
automated failure recovery. The suite.rc listing and accompanying
graph in Figure~\ref{fig-suicide} show how to define a chain of failure
recovery tasks
that trigger if they're needed but otherwise remove themselves from the
suite (you can run the {\em AutoRecover.async} example suite to see how
this works). The dashed graph edges ending in solid dots indicate
suicide triggers, and the open arrowheads indicate conditional triggers
as usual. Suicide triggers are ignored by default in the graph view, unless you toggle them on with {\em View} -> {\em Options} -> {\em Ignore Suicide Triggers}.

\begin{figure}
\begin{minipage}[b]{0.5\textwidth}
\lstset{language=suiterc}
\begin{lstlisting}
[meta]
    title = automated failure recovery
    description = """
Model task failure triggers diagnosis
and recovery tasks, which take themselves
out of the suite if model succeeds. Model
post processing triggers off model OR
recovery tasks.
              """
[scheduling]
    [[dependencies]]
        graph = """
pre => model
model:fail => diagnose => recover
model => !diagnose & !recover
model | recover => post
                """
[runtime]
    [[model]]
        # UNCOMMENT TO TEST FAILURE:
        # script = /bin/false
\end{lstlisting}
\lstset{language=transcript}
\end{minipage}
\hfill
\begin{minipage}[b]{0.5\textwidth}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{graphics/png/orig/suicide.png}
    \end{center}
\end{minipage}
\caption[Automated failure recovery via suicide triggers] {\scriptsize
Automated failure recovery via suicide triggers.}
\label{fig-suicide}
\end{figure}

Note that multiple suicide triggers combine in the same way as other triggers, so this:
\begin{lstlisting}
foo => !baz
bar => !baz
\end{lstlisting}
is equivalent to this:
\begin{lstlisting}
foo & bar => !baz
\end{lstlisting}
i.e.\ both \lstinline=foo= and \lstinline=bar= must succeed for
\lstinline=baz= to be taken out of the suite. If you really want a task
to be taken out if any one of several events occurs then be careful to
write it that way:
\begin{lstlisting}
foo | bar => !baz
\end{lstlisting}

A word of warning on the meaning of ``bare suicide triggers''. Consider
the following suite:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => !bar"
\end{lstlisting}
Task \lstinline=bar= has a suicide trigger but no normal prerequisites
(a suicide trigger is not a task triggering prerequisite, it is a task
removal prerequisite) so this is entirely equivalent to:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
            foo & bar
           foo => !bar
                """
\end{lstlisting}
In other words both tasks will trigger immediately, at the same time,
and then \lstinline=bar= will be removed if \lstinline=foo= succeeds.

If an active task proxy (currently in the submitted or running states)
is removed from the suite by a suicide trigger, a warning will be logged.

\paragraph{Family Triggers}
\label{FamilyTriggers}

Families defined by the namespace inheritance hierarchy
(~\ref{NIORP}) can be used in the graph trigger whole groups of
tasks at the same time (e.g.\ forecast model ensembles and groups of
tasks for processing different observation types at the same time) and
for triggering downstream tasks off families as a whole. Higher level
families, i.e.\ families of families, can also be used, and are reduced
to the lowest level member tasks. Note that tasks can also trigger off
individual family members if necessary.

To trigger an entire task family at once:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => FAM"
[runtime]
    [[FAM]]    # a family (because others inherit from it)
    [[m1,m2]]  # family members (inherit from namespace FAM)
        inherit = FAM
\end{lstlisting}
This is equivalent to:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "foo => m1 & m2"
[runtime]
    [[FAM]]
    [[m1,m2]]
        inherit = FAM
\end{lstlisting}

To trigger other tasks off families we have to specify whether
to triggering off {\em all members} starting, succeeding, failing,
or finishing, or off {\em any} members (doing the same). Legal family
triggers are thus:

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
      # all-member triggers:
    FAM:start-all => one
    FAM:succeed-all => one
    FAM:fail-all => one
    FAM:finish-all => one
      # any-member triggers:
    FAM:start-any => one
    FAM:succeed-any => one
    FAM:fail-any => one
    FAM:finish-any => one
                """
\end{lstlisting}

Here's how to trigger downstream processing after if one or more family
members succeed, but only after all members have finished (succeeded or
failed):

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
    FAM:finish-all & FAM:succeed-any => foo
                """
\end{lstlisting}

\paragraph{Writing Efficient Inter-Family Triggering}
\label{EfficientInterFamilyTriggering}

While cylc allows writing dependencies between two families it is important to
consider the number of dependencies this will generate. In the following
example, each member of \lstinline=FAM2= has dependencies pointing at all the
members of \lstinline=FAM1=.

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
    FAM1:succeed-any => FAM2
                """
\end{lstlisting}

Expanding this out, you generate \lstinline=N * M= dependencies, where
\lstinline=N= is the number of members of \lstinline=FAM1= and \lstinline=M= is
the number of members of \lstinline=FAM2=. This can result in high memory use
as the number of members of these families grows, potentially rendering the
suite impractical for running on some systems.

You can greatly reduce the number of dependencies generated in these situations
by putting dummy tasks in the graphing to represent the state of the family you
want to trigger off. For example, if \lstinline=FAM2= should trigger off any
member of \lstinline=FAM1= succeeding you can create a dummy task
\lstinline=FAM1_succeed_any_marker= and place a dependency on it as follows:

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
    FAM1:succeed-any => FAM1_succeed_any_marker => FAM2
                """
[runtime]
...
    [[FAM1_succeed_any_marker]]
        script = true
...
\end{lstlisting}

This graph generates only \lstinline=N + M= dependencies, which takes
significantly less memory and CPU to store and evaluate.

\paragraph{Inter-Cycle Triggers}
\label{InterCyclePointTriggers}

Typically most tasks in a suite will trigger off others in the same
cycle point, but some may depend on others with other cycle points.
This notably applies to warm-cycled forecast models, which depend on
their own previous instances (see below); but other kinds of inter-cycle
dependence are possible too.\footnote{In NWP forecast analysis
suites parts of the observation processing and data assimilation
subsystem will typically also depend on model background fields
generated by the previous forecast.} Here's how to express this
kind of relationship in cylc:
\begin{lstlisting}
[dependencies]
    [[PT6H]]
        # B triggers off A in the previous cycle point
        graph = "A[-PT6H] => B"
\end{lstlisting}
inter-cycle and trigger type (or message trigger) notation can be
combined:
\begin{lstlisting}
    # B triggers if A in the previous cycle point fails:
    graph = "A[-PT6H]:fail => B"
\end{lstlisting}

At suite start-up inter-cycle triggers refer to a previous cycle point
that does not exist. This does not cause the dependent task to wait
indefinitely, however, because cylc ignores triggers that reach back
beyond the initial cycle point. That said, the presence of an
inter-cycle trigger does normally imply that something special has to
happen at start-up. If a model depends on its own previous instance for
restart files, for instance, then an initial set of restart files has to be
generated somehow or the first model task will presumably fail with
missing input files. There are several ways to handle this in cylc
using different kinds of one-off (non-cycling) tasks that run at suite
start-up. They are illustrated in the Tutorial
(\ref{TutInterCyclePointTriggers}); to summarize here briefly:

\begin{myitemize}
    \item \lstinline=R1= tasks (recommended):
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[R1]]]
            graph = "prep"
        [[[R1/T00,R1/T12]]]
            graph = "prep[^] => foo"
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}

\end{myitemize}
\lstset{language=transcript}

\lstinline=R1=, or \lstinline=R1/date-time= tasks are the recommended way to
specify unusual start up conditions. They allow you to specify a clean
distinction between the dependencies of initial cycles and the dependencies
of the subsequent cycles.

Initial tasks can be used for real model cold-start processes, whereby a
warm-cycled model at any given cycle point can in principle have its inputs
satisfied by a previous instance of itself, {\em or} by an initial task with
(nominally) the same cycle point.

In effect, the \lstinline=R1= task masquerades as the previous-cycle-point trigger
of its associated cycling task. At suite start-up initial tasks will
trigger the first cycling tasks, and thereafter the inter-cycle trigger
will take effect.

If a task has a dependency on another task in a different cycle point, the
dependency can be written using the \lstinline=[offset]= syntax such as
\lstinline=[-PT12H]= in \lstinline@foo[-PT12H] => foo@. This means that
\lstinline=foo= at the current cycle point depends on a previous instance of
 \lstinline=foo= at 12 hours before the current cycle point. Unlike the
 cycling section headings (e.g.\ \lstinline=[[[T00,T12]]]=), dependencies
 assume that relative times are relative to the current cycle point, not the
 initial cycle point.

However, it can be useful to have specific dependencies on tasks at or near
the initial cycle point. You can switch the context of the offset to be
the initial cycle point by using the caret symbol: \lstinline=^=.

For example, you can write \lstinline=foo[^]= to mean foo at the initial
cycle point, and \lstinline=foo[^+PT6H]= to mean foo 6 hours after the initial
cycle point. Usually, this kind of dependency will only apply in a limited
number of cycle points near the start of the suite, so you may want to write
it in \lstinline=R1=-based cycling sections. Here's the example inter-cycle
\lstinline=R1= suite from above again.

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[R1]]]
            graph = "prep"
        [[[R1/T00,R1/T12]]]
            graph = "prep[^] => foo"
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}
\lstset{language=transcript}

You can see there is a dependence on the initial \lstinline=R1= task
\lstinline=prep= for \lstinline=foo= at the first \lstinline=T00= cycle point,
and at the first \lstinline=T12= cycle point. Thereafter, \lstinline=foo= just
depends on its previous (12 hours ago) instance.

Finally, it is also possible to have a dependency on a task at a specific cycle
point.

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[R1/20200202]]]
            graph = "baz[20200101] => qux"
\end{lstlisting}
\lstset{language=transcript}

However, in a long running suite, a repeating cycle should avoid having a
dependency on a task with a specific cycle point (including the initial cycle
point) - as it can currently cause performance issue. In the following example,
all instances of \lstinline=qux= will depend on \lstinline=baz.20200101=, which
will never be removed from the task pool.:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 2010
    [[dependencies]]
        # Can cause performance issue!
        [[[P1D]]]
            graph = "baz[20200101] => qux"
\end{lstlisting}
\lstset{language=transcript}

\paragraph{Special Sequential Tasks}
\label{SequentialTasks}

If a cycling task does not generate files required by its own successor,
then successive instances can run in parallel if the opportunity arises.
However, if such a task would interfere with its own siblings for
internal reasons (e.g.\ use of a hardwired non cycle dependent
temporary file or similar) then it can be forced to run sequentially.
This can be done with explicit inter-cycle triggers in the graph:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T12]]]
            graph = "foo[-PT12H] => foo => bar"
\end{lstlisting}
or by declaring the task to be {\em sequential}:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[special tasks]]
        sequential = foo
    [[dependencies]]
        [[[T00,T12]]]
            graph = "foo => bar"
\end{lstlisting}

The {\em sequential} declaration also results in each instance of
\lstinline=foo= triggering off its own predecessor, exactly as in
the explicit version. The only difference is that implicit triggers will
not appear in graph visualizations. The implicit version can also be
considerably simpler when the task appears in multiple graph sections or
in a non-uniform cycling sequence: this suite:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[special tasks]]
        sequential = foo
    [[dependencies]]
        [[[T00,T03,T11]]]
            graph = "foo => bar"
\end{lstlisting}
is equivalent to this one:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T03,T11]]]
            graph = "foo => bar"
        [[[T00]]]
            graph = "foo[-PT13H] => foo"
        [[[T03]]]
            graph = "foo[-PT3H] => foo"
        [[[T11]]]
            graph = "foo[-PT8H] => foo"
\end{lstlisting}


\paragraph{Future Triggers}

Cylc also supports inter-cycle triggering off tasks ``in the future'' (with
respect to cycle point):
\begin{lstlisting}
[[dependencies]]
    [[[T00,T06,T12,T18]]]
        graph = """
    # A runs in this cycle:
            A
    # B in this cycle triggers off A in the next cycle.
            A[PT6H] => B
        """
\end{lstlisting}
(Recall that \lstinline=A[t+PT6H]= can run before B[t] because tasks in cylc
have private cycle points). Future triggers present a problem at the suite
shutdown rather than at start-up. Here, \lstinline=B= at the final cycle
point wants to trigger off an instance of \lstinline=A= that will never exist
because it is beyond the suite stop point. Consequently cylc prevents tasks
from spawning successors that depend on other tasks beyond the stop point.

\paragraph{Clock Triggers}
\label{ClockTriggerTasks}

In addition to depending on other tasks (and on external events -
see~\ref{ExternalTriggers}) tasks can depend on the wall clock: specifically,
they can trigger off a wall clock time expressed as an offset from their own
cycle point:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[special tasks]]
        clock-trigger = foo(PT2H)
    [[dependencies]]
        [[[T00]]]
            graph = foo
\end{lstlisting}
Here, \lstinline=foo[2015-08-23T00]= would trigger (other dependencies allowing)
when the wall clock time reaches \lstinline=2015-08-23T02=. Clock-trigger
offsets are normally positive, to trigger some time {\em after} the wall-clock
time is equal to task cycle point.

Clock-triggers have no effect on scheduling if the suite is running sufficiently
far behind the clock (e.g.\ after a delay, or because it is processing archived
historical data) that the trigger times, which are relative to task cycle
point, have already passed.

\paragraph{Clock-Expire Triggers}
\label{ClockExpireTasks}

Tasks can be configured to {\em expire} - i.e.\ to skip job submission and
enter the {\em expired} state - if they are too far behind the wall clock when
they become ready to run, and other tasks can trigger off this. As a possible
use case, consider a cycling task that copies the latest of a set of files to
overwrite the previous set: if the task is delayed by more than one cycle there
may be no point in running it because the freshly copied files will just be
overwritten immediately by the next task instance as the suite catches back up
to real time operation. Clock-expire tasks are configured like clock-trigger
tasks, with a date-time offset relative to cycle point (\ref{ClockExpireRef}).
The offset should be positive to make the task expire if the wall-clock time
has gone beyond the cycle point. Triggering off an expired task typically
requires suicide triggers to remove the workflow that runs if the task has not
expired. Here a task called \lstinline=copy= expires, and its downstream
workflow is skipped, if it is more than one day behind the wall-clock (see also
\lstinline=examples/clock-expire=):
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
   cycle point format = %Y-%m-%dT%H
[scheduling]
    initial cycle point = 2015-08-15T00
    [[special tasks]]
        clock-expire = copy(-P1D)
    [[dependencies]]
        [[[P1D]]]
            graph = """
        model[-P1D] => model => copy => proc
              copy:expired => !proc"""
\end{lstlisting}

\paragraph{External Triggers}
\label{ExternalTriggers}

In addition to depending on other tasks (and on the wall clock -
see~\ref{ClockTriggerTasks}) tasks can trigger off events reported by an
external system. For example, an external process could detect incoming data
on an ftp server, and then notify a suite containing a task to retrieve the
new data for processing. This is an alternative to long-running tasks that poll
for external events.

Note that cylc does not currently support triggering off ``filesystem events''
(e.g.\ \lstinline=inotify= on Linux). However, external watcher processes can
use filesystem events to detect triggering conditions, if that is appropriate,
before notifying a suite with our general external event system.

The external triggering process must call \lstinline=cylc ext-trigger= with the
name of the target suite, the message that identifies this type of event to the
suite, and an ID that distinguishes this particular event instance from others
(the name of the target task or its current cycle point is not required). The
event ID is just an arbitary string to cylc, but it typically identifies the
filename(s) of the latest dataset in some way. When the suite server program
receives the external event notification it will trigger the next instance of
any task waiting on that trigger (whatever its cycle point) and then broadcast
(see~\ref{cylc-broadcast}) the event ID to the cycle point of the triggered
task as \lstinline=$CYLC_EXT_TRIGGER_ID=. Downstream tasks with the same cycle
point therefore know the new event ID too and can use it, if they need to, to
identify the same new dataset. In this way a whole workflow can be associated
with each new dataset, and multiple datasets can be processed in parallel if
they happen to arrive in quick succession.

An externally-triggered task must register the event it waits on in the suite
scheduling section:
\lstset{language=suiterc}
\begin{lstlisting}
# suite "sat-proc"
[scheduling]
    cycling mode = integer
    initial cycle point = 1
    [[special tasks]]
        external-trigger = get-data("new sat X data avail")
    [[dependencies]]
        [[[P1]]]
            graph = get-data => conv-data => products
\end{lstlisting}

Then, each time a new dataset arrives the external detection system should
notify the suite like this:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc ext-trigger sat-proc "new sat X data avail" passX12334a
\end{lstlisting}
where ``sat-proc'' is the suite name and ``passX12334a'' is the ID string for
the new event. The suite passphrase must be installed on triggering account.

Note that only one task in a suite can trigger off a particular external
message. Other tasks can trigger off the externally triggered task as required,
of course.

\lstinline=$CYLC_DIR/examples/satellite/ext-triggers/suite.rc= is a working
example of a simulated satellite processing suite.

External triggers are not normally needed in date-time cycling suites driven
by real time data that comes in at regular intervals. In these cases a data
retrieval task can be clock-triggered (and have appropriate retry intervals
supplied) to submit at the expected data arrival time, so little time if any
is wasted in polling. However, if the arrival time of the cycle-point-specific
data is highly variable, external triggering may be used with the cycle point
embedded in the message:
\lstset{language=suiterc}
\begin{lstlisting}
# suite "data-proc"
[scheduling]
    initial cycle point = 20150125T00
    final cycle point   = 20150126T00
    [[special tasks]]
        external-trigger = get-data("data arrived for $CYLC_TASK_CYCLE_POINT")
    [[dependencies]]
        [[[T00]]]
            graph = init-process => get-data => post-process
\end{lstlisting}

Once the variable-length waiting is finished, an external detection system
should notify the suite like this:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc ext-trigger data-proc "data arrived for 20150126T00" passX12334a
\end{lstlisting}
where ``data-proc'' is the suite name, the cycle point has replaced the
variable in the trigger string, and ``passX12334a'' is the ID string for
the new event. The suite passphrase must be installed on the triggering
account. In this case, the event will trigger for the second cycle point but
not the first because of the cycle-point matching.

\subsubsection{Model Restart Dependencies}
\label{ModelRestartDependencies}

Warm-cycled forecast models generate {\em restart files}, e.g.\ model
background fields, to initialize the next forecast. This kind of
dependence requires an inter-cycle trigger:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            graph = "A[-PT6H] => A"
\end{lstlisting}

If your model is configured to write out additional restart files
to allow one or more cycle points to be skipped in an emergency {\em do not
represent these potential dependencies in the suite graph} as they
should not be used under normal circumstances. For example, the
following graph would result in task \lstinline=A= erroneously
triggering off \lstinline=A[T-24]= as a matter of course, instead of
off \lstinline=A[T-6]=, because \lstinline=A[T-24]= will always
be finished first:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00,T06,T12,T18]]]
            # DO NOT DO THIS (SEE ACCOMPANYING TEXT):
            graph = "A[-PT24H] | A[-PT18H] | A[-PT12H] | A[-PT6H] => A"
\end{lstlisting}

\subsubsection{How The Graph Determines Task Instantiation}

A graph trigger pair like \lstinline@foo => bar@ determines the existence and
prerequisites (dependencies) of the downstream task \lstinline=bar=, for
the cycle points defined by the associated graph section heading. In general it
does not say anything about the dependencies or existence of the upstream task
\lstinline=foo=. However {\em if the trigger has no cycle point offset} Cylc
will infer that \lstinline=bar= must exist at the same cycle points as
\lstinline=foo=. This is a convenience to allow this:

\lstset{language=suiterc}
\begin{lstlisting}
graph = "foo => bar"
\end{lstlisting}

to be written as shorthand for this:

\lstset{language=suiterc}
\begin{lstlisting}
graph = """foo
           foo => bar"""
\end{lstlisting}

(where \lstinline=foo= by itself means \lstinline@<nothing> => foo@, i.e.\ the
task exists at these cycle points but has no prerequisites - although other
prerequisites may be defined for it in other parts of the graph).

{\em Cylc does not infer the existence of the upstream task in offset
triggers} like \lstinline@foo[-P1D] => bar@ because, as explained in
Section~\ref{cylc-6-migration-implicit-cycling}, a typo in the offset interval
should generate an error rather than silently creating tasks on an erroneous
cycling sequence.

As a result you need to be careful not to define inter-cycle dependencies that
cannot be satisfied at run time. Suite validation catches this kind of error if
the existence of the cycle offset task is not defined anywhere at all:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020
    [[dependencies]]
        [[[P1Y]]]
            # ERROR
            graph = "foo[-P1Y] => bar"
\end{lstlisting}

\lstset{language=transcript}
\begin{lstlisting}
$ cylc validate SUITE
'ERROR: No cycling sequences defined for foo'
\end{lstlisting}

To fix this, use another line in the graph to tell Cylc to define
\lstinline=foo= at each cycle point:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020
    [[dependencies]]
        [[[P1Y]]]
            graph = """
                foo
                foo[-P1Y] => bar"""
\end{lstlisting}

But validation does not catch this kind of error if the offset task
is defined only on a different cycling sequence:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020
    [[dependencies]]
        [[[P2Y]]]
            graph = """foo
                # ERROR
                foo[-P1Y] => bar"""
\end{lstlisting}

This suite will validate OK, but it will stall at runtime with \lstinline=bar=
waiting on \lstinline=foo[-P1Y]= at the intermediate years where it does not
exist. The offset \lstinline=[-P1Y]= is presumably an error (it should be
\lstinline=[-P2Y]=), or else another graph line is needed to generate
\lstinline=foo= instances on the yearly sequence:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020
    [[dependencies]]
        [[[P1Y]]]
            graph = "foo"
        [[[P2Y]]]
            graph = "foo[-P1Y] => bar"
\end{lstlisting}

Similarly the following suite will validate OK, but it will stall at
runtime with \lstinline=bar= waiting on \lstinline=foo[-P1Y]= in
every cycle point, when only a single instance of it exists, at the initial
cycle point:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020
    [[dependencies]]
        [[[R1]]]
            graph = foo
        [[[P1Y]]]
            # ERROR
            graph = foo[-P1Y] => bar
\end{lstlisting}

Note that \lstinline=cylc graph= will display un-satisfiable inter-cycle
dependencies as ``ghost nodes''. Figure \ref{ghost-node-screenshot} is a
screenshot of cylc graph displaying the above example with the un-satisfiable
task (foo) displayed as a ``ghost node''.

\begin{figure}
    \begin{center}
        \includegraphics[width=0.6\textwidth]{graphics/png/orig/ghost-node-example.png}
    \end{center}
    \caption{Screenshot of \lstinline=cylc graph= showing one task as a
    ``ghost node''}
    \label{ghost-node-screenshot}
\end{figure}

\subsection{Runtime - Task Configuration}
\label{NIORP}

The \lstinline=[runtime]= section of a suite definition configures what
to execute (and where and how to execute it) when each task is ready to
run, in a {\em multiple inheritance hierarchy} of {\em
namespaces} culminating in individual tasks. This allows all common
configuration detail to be factored out and defined in one place.

Any namespace can configure any or all of the items defined in the
{\em Suite.rc Reference} (\ref{SuiteRCReference}).

Namespaces that do not explicitly inherit from others automatically
inherit from the {\em root} namespace (below).

Nested namespaces define {\em task families} that can be used in the
graph as convenient shorthand for triggering all member tasks at once,
or for triggering other tasks off all members at once -
see~\ref{FamilyTriggers}. Nested namespaces can be
progressively expanded and collapsed in the dependency graph viewer, and
in the gcylc graph and text views. Only the first parent of each
namespace (as for single-inheritance) is used for suite visualization
purposes.

\subsubsection{Namespace Names}

Namespace names may contain letters, digits, underscores, and hyphens.

Note that {\em task names need not be hardwired into task implementations}
because task and suite identity can be extracted portably from the task
execution environment supplied by the suite server program
(\ref{TaskExecutionEnvironment}) - then to rename a task you can just change
its name in the suite definition.

\subsubsection{Root - Runtime Defaults}

The root namespace, at the base of the inheritance hierarchy,
provides default configuration for all tasks in the suite.
Most root items are unset by default, but some have default values
sufficient to allow test suites to be defined by dependency graph alone.
The {\em script} item, for example, defaults to code that
prints a message then sleeps for between 1 and 15 seconds and
exits. Default values are documented with each item in~\ref{SuiteRCReference}.
You can override the defaults or
provide your own defaults by explicitly configuring the root namespace.

\subsubsection{Defining Multiple Namespaces At Once}
\label{MultiTaskDef}

If a namespace section heading is a comma-separated list of names
then the subsequent configuration applies to each list member.
Particular tasks can be singled out at run time using the
\lstinline=$CYLC_TASK_NAME= variable.

As an example, consider a suite containing an ensemble of closely
related tasks that each invokes the same script but with a unique
argument that identifies the calling task name:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[ENSEMBLE]]
        script = "run-model.sh $CYLC_TASK_NAME"
    [[m1, m2, m3]]
        inherit = ENSEMBLE
\end{lstlisting}

For large ensembles Jinja2 template processing can be used to
automatically generate the member names and associated dependencies
(see~\ref{Jinja2}).

\subsubsection{Runtime Inheritance - Single}

The following listing of the {\em inherit.single.one} example suite
illustrates basic runtime inheritance with single parents.

\lstset{language=suiterc}
\lstinputlisting{../../../examples/inherit/single/one/suite.rc}
\lstset{language=transcript}

\subsubsection{Runtime Inheritance - Multiple}

If a namespace inherits from multiple parents the linear order of
precedence (which namespace overrides which) is determined by the
so-called {\em C3 algorithm} used to find the linear {\em method
resolution order} for class hierarchies in Python and several other
object oriented programming languages. The result of this should be
fairly obvious for typical use of multiple inheritance in cylc suites,
but for detailed documentation of how the algorithm works refer to the
official Python documentation here:
\lstinline=http://www.python.org/download/releases/2.3/mro/=.

The {\em inherit.multi.one} example suite, listed here, makes use of
multiple inheritance:

\lstset{language=suiterc}
\lstinputlisting{../../../examples/inherit/multi/one/suite.rc}
\lstset{language=transcript}

\lstinline=cylc get-suite-config= provides an easy way to check the result of
inheritance in a suite. You can extract specific items, e.g.:
\begin{lstlisting}
$ cylc get-suite-config --item '[runtime][var_p2]script' \
    inherit.multi.one
echo ``RUN: run-var.sh''
\end{lstlisting}
or use the \lstinline=--sparse= option to print entire namespaces
without obscuring the result with the dense runtime structure obtained
from the root namespace:
\begin{lstlisting}
$ cylc get-suite-config --sparse --item '[runtime]ops_s1' inherit.multi.one
script = echo ``RUN: run-ops.sh''
inherit = ['OPS', 'SERIAL']
[directives]
   job_type = serial
\end{lstlisting}

\paragraph{Suite Visualization And Multiple Inheritance}

The first parent inherited by a namespace is also used as the
collapsible family group when visualizing the suite. If this is not what
you want, you can demote the first parent for visualization purposes,
without affecting the order of inheritance of runtime properties:
\begin{lstlisting}
[runtime]
    [[BAR]]
        # ...
    [[foo]]
        # inherit properties from BAR, but stay under root for visualization:
        inherit = None, BAR
\end{lstlisting}


\subsubsection{How Runtime Inheritance Works}

The linear precedence order of ancestors is computed for each namespace
using the C3 algorithm. Then any runtime items that are explicitly
configured in the suite definition are ``inherited'' up the linearized
hierarchy for each task, starting at the root namespace: if a particular
item is defined at multiple levels in the hierarchy, the level nearest
the final task namespace takes precedence. Finally, root namespace
defaults are applied for every item that has not been configured in the
inheritance process (this is more efficient than carrying the full dense
namespace structure through from root from the beginning).

\subsubsection{Task Execution Environment}
\label{TaskExecutionEnvironment}

The task execution environment contains suite and task identity variables
provided by the suite server program, and user-defined environment variables.
The environment is explicitly exported (by the task job script) prior to
executing the task \lstinline=script= (see~\ref{TaskJobSubmission}).

Suite and task identity are exported first, so that user-defined
variables can refer to them. Order of definition is preserved throughout
so that variable assignment expressions can safely refer to previously
defined variables.

Additionally, access to cylc itself is configured prior to the user-defined
environment, so that variable assignment expressions can make use of
cylc utility commands:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        [[[environment]]]
            REFERENCE_TIME = $( cylc util cycletime --offset-hours=6 )
\end{lstlisting}

\paragraph{User Environment Variables}

A task's user-defined environment results from its inherited
\lstinline=[[[environment]]]= sections:
\begin{lstlisting}
[runtime]
    [[root]]
        [[[environment]]]
            COLOR = red
            SHAPE = circle
    [[foo]]
        [[[environment]]]
            COLOR = blue  # root override
            TEXTURE = rough # new variable
\end{lstlisting}
This results in a task {\em foo} with
\lstinline@SHAPE=circle@,
\lstinline@COLOR=blue@, and
\lstinline@TEXTURE=rough@ in its environment.

\paragraph{Overriding Environment Variables}

When you override inherited namespace items the original parent
item definition is {\em replaced} by the new definition. This applies to
all items including those in the environment sub-sections which,
strictly speaking, are not ``environment variables'' until they are
written, post inheritance processing, to the task job script that
executes the associated task. Consequently, if you override an
environment variable you cannot also access the original parent value:
\begin{lstlisting}
[runtime]
    [[FOO]]
        [[[environment]]]
            COLOR = red
    [[bar]]
        inherit = FOO
        [[[environment]]]
            tmp = $COLOR        # !! ERROR: $COLOR is undefined here
            COLOR = dark-$tmp   # !! as this overrides COLOR in FOO.
\end{lstlisting}
The compressed variant of this, \lstinline@COLOR = dark-$COLOR@, is
also in error for the same reason. To achieve the desired result you
must use a different name for the parent variable:
\begin{lstlisting}
[runtime]
    [[FOO]]
        [[[environment]]]
            FOO_COLOR = red
    [[bar]]
        inherit = FOO
        [[[environment]]]
            COLOR = dark-$FOO_COLOR  # OK
\end{lstlisting}

\paragraph{Task Job Script Variables}
\label{Task Job Script Variables}

These are variables that can be referenced (but should not be modified) in a
task job script.

The task job script may export the following environment variables:

\lstset{language=bash}
\begin{lstlisting}
CYLC_DEBUG                      # Debug mode, true or not defined
CYLC_DIR                        # Location of cylc installation used
CYLC_VERSION                    # Version of cylc installation used

CYLC_CYCLING_MODE               # Cycling mode, e.g. gregorian
CYLC_SUITE_FINAL_CYCLE_POINT    # Final cycle point
CYLC_SUITE_INITIAL_CYCLE_POINT  # Initial cycle point
CYLC_SUITE_NAME                 # Suite name
CYLC_UTC                        # UTC mode, True or False
CYLC_VERBOSE                    # Verbose mode, True or False
TZ                              # Set to "UTC" in UTC mode or not defined

CYLC_SUITE_RUN_DIR              # Location of the suite run directory in
                                # job host, e.g. ~/cylc-run/foo
CYLC_SUITE_DEF_PATH             # Location of the suite definition directory in
                                # job host, e.g. ~/cylc-run/foo
CYLC_SUITE_HOST                 # Host running the suite process
CYLC_SUITE_OWNER                # User ID running the suite process
CYLC_SUITE_DEF_PATH_ON_SUITE_HOST
                                # Location of the suite definition directory in
                                # suite host, e.g. ~/cylc-run/foo
CYLC_SUITE_SHARE_DIR            # Suite (or task!) shared directory (see below)
CYLC_SUITE_WORK_DIR             # Suite work directory (see below)

CYLC_TASK_JOB                   # Task job identifier expressed as
                                # CYCLE-POINT/TASK-NAME/SUBMIT-NUM
                                # e.g. 20110511T1800Z/t1/01
CYLC_TASK_CYCLE_POINT           # Cycle point, e.g. 20110511T1800Z
CYLC_TASK_NAME                  # Job's task name, e.g. t1
CYLC_TASK_SUBMIT_NUMBER         # Job's submit number, e.g. 1,
                                # increments with every submit
CYLC_TASK_TRY_NUMBER            # Number of execution tries, e.g. 1
                                # increments with automatic retry-on-fail
CYLC_TASK_ID                    # Task instance identifier expressed as
                                # TASK-NAME.CYCLE-POINT
                                # e.g. t1.20110511T1800Z
CYLC_TASK_LOG_ROOT              # Location of the job file
                                # e.g. ~/cylc-run/foo/log/job/20110511T1800Z/t1/01/job
CYLC_TASK_WORK_DIR              # Location of task work directory (see below)
                                # e.g. ~/cylc-run/foo/work/20110511T1800Z/t1
CYLC_TASK_NAMESPACE_HIERARCHY   # Linearised family namespace of the task,
                                # e.g. root postproc t1

CYLC_TASK_COMMS_METHOD          # Set to "ssh" if communication method is "ssh"
CYLC_TASK_SSH_LOGIN_SHELL       # With "ssh" communication, if set to "True",
                                # use login shell on suite host
\end{lstlisting}

There are also some global shell variables that may be defined in the task job
script (but not exported to the environment). These include:
\lstset{language=bash}
\begin{lstlisting}
CYLC_FAIL_SIGNALS               # List of signals trapped by the error trap
CYLC_VACATION_SIGNALS           # List of signals trapped by the vacation trap
CYLC_SUITE_WORK_DIR_ROOT        # Root directory above the suite work directory
                                # in the job host
CYLC_TASK_MESSAGE_STARTED_PID   # PID of "cylc message started" command
CYLC_TASK_WORK_DIR_BASE         # Alternate task work directory,
                                # relative to the suite work directory
\end{lstlisting}

\paragraph{Suite Share Directories}

A {\em suite share directory} is created automatically under the suite run
directory as a share space for tasks. The location is available to tasks as
\lstinline=$CYLC_SUITE_SHARE_DIR=. In a cycling suite, output files are
typically held in cycle point sub-directories of the suite share directory.

The top level share and work directory (below) location can be changed
(e.g.\ to a large data area) by a global config setting
(see~\ref{workdirectory}).

\paragraph{Task Work Directories}

Task job scripts are executed from within {\em work directories} created
automatically under the suite run directory. A task can get its own work
directory from \lstinline=$CYLC_TASK_WORK_DIR= (or simply \lstinline=$PWD= if
it does not \lstinline=cd= elsewhere at runtime). By default the location
contains task name and cycle point, to provide a unique workspace for every
instance of every task. This can be overridden in the suite definition,
however, to get several tasks to share the same work directory
(see~\ref{worksubdirectory}).

The top level work and share directory (above) location can be changed
(e.g.\ to a large data area) by a global config setting
(see~\ref{workdirectory}).

\lstset{language=transcript}

\paragraph{Environment Variable Evaluation}

Variables in the task execution environment are not evaluated in the
shell in which the suite is running prior to submitting the task. They
are written in unevaluated form to the job script that is submitted by
cylc to run the task (\ref{JobScripts}) and are therefore
evaluated when the task begins executing under the task owner account
on the task host. Thus \lstinline=$HOME=, for instance, evaluates at
run time to the home directory of task owner on the task host.

\subsubsection{How Tasks Get Access To The Suite Directory}

Tasks can use \lstinline=$CYLC_SUITE_DEF_PATH= to access suite files on
the task host, and the suite bin directory is automatically added
\lstinline=$PATH=. If a remote suite definition directory is not
specified the local (suite host) path will be assumed with the local
home directory, if present, swapped for literal \lstinline=$HOME= for
evaluation on the task host.

\subsubsection{Remote Task Hosting}
\label{RunningTasksOnARemoteHost}

If a task declares an owner other than the suite owner and/or
a host other than the suite host, cylc will use non-interactive ssh to
execute the task on the \lstinline=owner@host= account by the configured
batch system:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        [[[remote]]]
            host = orca.niwa.co.nz
            owner = bob
        [[[job]]]
            batch system = pbs
\end{lstlisting}
\lstset{language=transcript}
For this to work:
\begin{myitemize}
    \item non-interactive ssh is required from the suite host to the remote
    task accounts.

    \item cylc must be installed on task hosts.
    \begin{myitemize}
        \item Optional software dependencies such as graphviz and
        Jinja2 are not needed on task hosts.
        \item If polling task communication is used, there is no other
        requirement.
        \item If SSH task communication is configured, non-interactive ssh is
        required from the task host to the suite host.
        \item If (default) task communication is configured, the task host
        should have access to the port on the suite host.
    \end{myitemize}
    \item the suite definition directory, or some fraction of its
        content, can be installed on the task host, if needed.
\end{myitemize}

To learn how to give remote tasks access to cylc,
see~\ref{HowTasksGetAccessToCylc}.

Tasks running on the suite host under another user account are treated as
remote tasks.

Remote hosting, like all namespace settings, can be declared globally in
the root namespace, or per family, or for individual tasks.

\paragraph{Dynamic Host Selection}

Instead of hardwiring host names into the suite definition you can
specify a shell command that prints a hostname, or an environment
variable that holds a hostname, as the value of the host config item.
See~\ref{DynamicHostSelection}.

\paragraph{Remote Task Log Directories}

Task stdout and stderr streams are written to log files in a
suite-specific sub-directory of the {\em suite run directory}, as
explained in~\ref{WhitherStdoutAndStderr}. For remote tasks
the same directory is used, but {\em on the task host}.
Remote task log directories, like local ones, are created on the fly, if
necessary, during job submission.

\subsection{Visualization}
\label{viso}

The visualization section of a suite definition is used to configure
suite graphing, principally graph node (task) and edge (dependency
arrow) style attributes. Tasks can be grouped for the purpose of
applying common style attributes. See~\ref{SuiteRCReference} for details.

\subsubsection{Collapsible Families In Suite Graphs}

\lstset{language=suiterc}
\begin{lstlisting}
[visualization]
    collapsed families = family1, family2
\end{lstlisting}
\lstset{language=transcript}

Nested families from the runtime inheritance hierarchy can be expanded
and collapsed in suite graphs and the gcylc graph view. All families
are displayed in the collapsed state at first, unless
\lstinline=[visualization]collapsed families= is used to single out
specific families for initial collapsing.

In the gcylc graph view, nodes outside of the main graph (such as the
members of collapsed families) are plotted as rectangular nodes to
the right if they are doing anything interesting (submitted, running,
failed).

Figure~\ref{fig-namespaces} illustrates successive expansion of nested task
families in the {\em namespaces} example suite.

\begin{figure}
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-2.png}
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-3.png}
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-4.png}
    \end{center}
\end{minipage}

\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-5.png}
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-6.png}
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.3\textwidth}
    \begin{center}
        \includegraphics[width=\textwidth]{graphics/png/orig/inherit-7.png}
    \end{center}
\end{minipage}
\caption[{\em namespaces} example suite graphs]{\scriptsize Graphs of the {\em
namespaces} example suite showing various states of expansion of the
nested namespace family hierarchy, from all families collapsed (top
left) through to all expanded (bottom right). This can also be done by
right-clicking on tasks in the gcylc graph view.}
\label{fig-namespaces}
\end{figure}

\subsection{Parameterized Tasks}
\label{Parameterized Tasks}

Cylc can automatically generate tasks and dependencies by expanding
parameterized task names over lists of parameter values. Uses for this
include:
\begin{myitemize}
    \item generating an ensemble of similar model runs
    \item generating chains of tasks to process similar datasets
    \item replicating an entire workflow, or part thereof, over several runs
    \item splitting a long model run into smaller steps or ``chunks``
        (parameterized cycling)
\end{myitemize}

{\em Note that this can be done with Jinja2 loops too (Section~\ref{Jinja2})
    but parameterization is much cleaner (nested loops can seriously reduce
the clarity of a suite definition).}

\subsubsection{Parameter Expansion}

Parameter values can be lists of strings, or lists of integers and
integer ranges (with inclusive bounds). Numeric values in a list of strings are
considered strings. It is not possible to mix strings with integer ranges.

For example:

\begin{lstlisting}
[cylc]
    [[parameters]]
        # parameters: "ship", "buoy", "plane"
        # default task suffixes: _ship, _buoy, _plane
        obs = ship, buoy, plane

        # parameters: 1, 2, 3, 4, 5
        # default task suffixes: _run1, _run2, _run3, _run4, _run5
        run = 1..5

        # parameters: 1, 3, 5, 7, 9
        # default task suffixes: _idx1, _idx3, _idx5, _idx7, _idx9
        idx = 1..9..2

        # parameters: -11, -1, 9
        # default task suffixes: _idx-11, _idx-01, _idx+09
        idx = -11..9..10

        # parameters: 1, 3, 5, 10, 11, 12, 13
        # default task suffixes: _i01, _i03, _i05, _i10, _i11, _i12, _i13
        i = 1..5..2, 10, 11..13

        # parameters: "0", "1", "e", "pi", "i"
        # default task suffixes: _0, _1, _e, _pi, _i
        item = 0, 1, e, pi, i

        # ERROR: mix strings with int range
        p = one, two, 3..5
\end{lstlisting}
Then angle brackets denote use of these parameters throughout the suite
definition. For the values above, this parameterized name:
\begin{lstlisting}
    model<run>  # for run = 1..2
\end{lstlisting}
expands to these concrete task names:
\begin{lstlisting}
    model_run1, model_run2
\end{lstlisting}
and this parameterized name:
\begin{lstlisting}
    proc<obs>  # for obs = ship, buoy, plane
\end{lstlisting}
expands to these concrete task names:
\begin{lstlisting}
    proc_ship, proc_buoy, proc_plane
\end{lstlisting}
By default, to avoid any ambiguity, the parameter name appears in the expanded
task names for integer values, but not for string values. For example,
\lstinline=model_run1= for \lstinline@run = 1@, but \lstinline=proc_ship= for
\lstinline@obs = ship@. However, the default expansion templates can be
overridden if need be:
\begin{lstlisting}
[cylc]
    [[parameters]]
        obs = ship, buoy, plane
        run = 1..5
    [[parameter templates]]
        run = -R%(run)s  # Make foo<run> expand to foo-R1 etc.
\end{lstlisting}
(See~\ref{RefParameterTemplates} for more on the string template syntax.)

Any number of parameters can be used at once. This parameterization:
\begin{lstlisting}
    model<run,obs>  # for run = 1..2 and obs = ship, buoy, plane
\end{lstlisting}
expands to these tasks names:
\begin{lstlisting}
    model_run1_ship, model_run1_buoy, model_run1_plane,
    model_run2_ship, model_run2_buoy, model_run2_plane
\end{lstlisting}

Here's a simple but complete example suite:
\begin{lstlisting}
[cylc]
    [[parameters]]
        run = 1..2
[scheduling]
    [[dependencies]]
        graph = "prep => model<run>"
[runtime]
    [[model<run>]]
        # ...
\end{lstlisting}
The result, post parameter expansion, is this:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "prep => model_run1 & model_run2"
[runtime]
    [[model_run1]]
        # ...
    [[model_run2]]
        # ...
\end{lstlisting}

Here's a more complex graph using two parameters (\lstinline@[runtime]@ omitted):
\begin{lstlisting}
[cylc]
    [[parameters]]
        run = 1..2
        mem = cat, dog
[scheduling]
    [[dependencies]]
        graph = """prep => init<run> => model<run,mem> =>
                      post<run,mem> => wrap<run> => done"""
\end{lstlisting}
%which expands to:
%\begin{lstlisting}
%[scheduling]
%    [[dependencies]]
%        graph = """
%            prep => init_run1 => model_run1_cat => post_run1_cat => wrap_run1 => done
%                init_run1 => model_run1_dog => post_run2_dog => wrap_run1
%            prep => init_run2 => model_run2_cat => post_run2_cat => wrap_run2 => done
%                init_run2 => model_run2_dog => post_run2_dog => wrap_run2"""
%\end{lstlisting}
Figure~\ref{fig-params-1} shows the result as visualized by \lstinline=cylc graph=.
\begin{figure}
    \begin{center}
        \includegraphics[width=10cm]{graphics/png/orig/params1.png}
    \end{center}
    \caption[Parameter expansion example.]{\scriptsize
     Parameter expansion example.}
    \label{fig-params-1}
\end{figure}

\paragraph{Zero-Padded Integer Values}

Integer parameter values are given a default template for generating task
suffixes that are zero-padded according to the longest size of their values.
For example, the default template for \lstinline@p = 9..10@ would be
\lstinline@_p%(p)02d@, so that \lstinline@foo<p>@ would become
\lstinline@foo_p09, foo_p10@.
If negative values are present in the parameter list, the
default template will include the sign.
For example, the default template for \lstinline@p = -1..1@ would be
\lstinline@_p%(p)+02d@, so that \lstinline@foo<p>@ would become
\lstinline@foo_p-1, foo_p+0, foo_p+1@.

To get thicker padding and/or alternate suffixes, use a template. E.g.:

\begin{lstlisting}
[cylc]
    [[parameters]]
        i = 1..9
        p = 3..14
    [[parameter templates]]
        i = _i%(i)02d  # suffixes = _i01, _i02, ..., _i09
        # A double-percent gives a literal percent character
        p = %%p%(p)03d  # suffixes = %p003, %p004, ..., %p013, %p014
\end{lstlisting}

\paragraph{Parameters as Full Task Names}

Parameter values can be used as full task names, but the default template
should be overridden to remove the initial underscore.
For example:

\begin{lstlisting}
[cylc]
    [[parameters]]
        i = 1..4
        obs = ship, buoy, plane
    [[parameter templates]]
        i = i%(i)d  # task name must begin with an alphabet
        obs = %(obs)s
[scheduling]
    [[dependencies]]
        graph = """
foo => <i>  # foo => i1 & i2 & i3 & i4
<obs> => bar  # ship & buoy & plane => bar
"""
\end{lstlisting}

\subsubsection{Passing Parameter Values To Tasks}

Parameter values are passed as environment variables to tasks generated by
parameter expansion. For example, if we have:

\begin{lstlisting}
[cylc]
    [[parameters]]
        obs = ship, buoy, plane
        run = 1..5
[scheduling]
    [[dependencies]]
        graph = model<run,obs>
\end{lstlisting}

Then task \lstinline=model_run2_ship= would get the following standard
environment variables:

\begin{lstlisting}
# In a job script of an instance of the "model_run2_ship" task:
export CYLC_TASK_PARAM_run="2"
export CYLC_TASK_PARAM_obs="ship"
\end{lstlisting}

These variables allow tasks to determine which member of a parameterized
group they are, and so to vary their behaviour accordingly.

You can also define custom variables and string templates for parameter value
substitution. For example, if we add this to the above configuration:

\begin{lstlisting}
[runtime]
    [[model<run,obs>]]
        [[[parameter environment templates]]]
            MYNAME = %(obs)sy-mc%(obs)sface
            MYFILE = /path/to/run%(run)03d/%(obs)s
\end{lstlisting}

Then task \lstinline=model_run2_ship= would get the following custom
environment variables:

\begin{lstlisting}
# In a job script of an instance of the "model_run2_ship" task:
export MYNAME=shipy-mcshipface
export MYFILE=/path/to/run002/ship
\end{lstlisting}

\subsubsection{Selecting Specific Parameter Values}

Specific parameter values can be singled out in the graph and under
\lstinline=[runtime]= with the notation \lstinline@<p=5>@ (for example).
Here's how to make a special task trigger off just the first of a
set of model runs:
\begin{lstlisting}
[cylc]
    [[parameters]]
        run = 1..5
[scheduling]
    [[dependencies]]
        graph = """model<run> => post_proc<run>  # general case
                   model<run=1> => check_first_run  # special case"""
[runtime]
    [[model<run>]]
        # config for all "model" runs...
    [[model<run=1>
        # special config (if any) for the first model run...
    #...
\end{lstlisting}

\subsubsection{Selecting Partial Parameter Ranges}

The parameter notation does not currently support partial range selection such
as \lstinline@foo<p=5..10>@, but you can achieve the same result by defining a
second parameter that covers the partial range and giving it the same expansion
template as the full-range parameter. For example:

\begin{lstlisting}
[cylc]
    [[parameters]]
        run = 1..10  # 1, 2, ..., 10
        runx = 1..3  # 1, 2, 3
    [[parameter templates]]
        run = _R%(run)02d   # _R01, _R02, ..., _R10
        runx = _R%(runx)02d  # _R01, _R02, _R03
[scheduling]
    [[dependencies]]
        graph = """model<run> => post<run>
                   model<runx> => checkx<runx>"""
[runtime]
    [[model<run>]]
        # ...
    #...
\end{lstlisting}


\subsubsection{Parameter Offsets In The Graph}

A negative offset notation \lstinline@<NAME-1>@ is interpreted as the previous
value in the ordered list of parameter values, while a positive offset is
interpreted as the next value. For example, to split a model run into multiple
steps with each step depending on the previous one, either of these graphs:
\begin{lstlisting}
    graph = "model<run-1> => model<run>"  # for run = 1, 2, 3
    graph = "model<run> => model<run+1>"  # for run = 1, 2, 3
\end{lstlisting}
expands to:
\begin{lstlisting}
    graph = """model_run1 => model_run2
               model_run2 => model_run3"""
# or equivalently:
    graph = "model_run1 => model_run2 => model_run3"
\end{lstlisting}
And this graph:
\begin{lstlisting}
    graph = "proc<size-1> => proc<size>"  # for size = small, big, huge
\end{lstlisting}
expands to:
\begin{lstlisting}
    graph = """proc_small => proc_big
               proc_big => proc_huge"""
# or equivalently:
    graph = "proc_small => proc_big => proc_huge"
\end{lstlisting}

However, a quirk in the current system means that you should avoid mixing
conditional logic in these statements. For example, the following will do the
unexpected:

\begin{lstlisting}
    graph = foo<m-1> & baz => foo<m>  # for m = cat, dog
\end{lstlisting}
currently expands to:
\begin{lstlisting}
    graph = foo_cat & baz => foo_dog
# when users may expect it to be:
#    graph = foo_cat => foo_dog
#    graph = baz => foo_cat & foo_dog
\end{lstlisting}

For the time being, writing out the logic explicitly will give you the correct
graph.

\begin{lstlisting}
    graph = """foo<m-1> => foo<m>  # for m = cat, dog
               baz => foo<m>"""
\end{lstlisting}

\subsubsection{Task Families And Parameterization}

Task family members can be generated by parameter expansion:
\begin{lstlisting}
[runtime]
    [[FAM]]
    [[member<r>]]
        inherit = FAM
# Result: family FAM contains member_r1, member_r2, etc.
\end{lstlisting}

Family names can be parameterized too, just like task names:
\begin{lstlisting}
[runtime]
    [[RUN<r>]]
    [[model<r>]]
        inherit = RUN<r>
    [[post_proc<r>]]
        inherit = RUN<r>
# Result: family RUN_r1 contains model_r1 and post_proc_r1,
#         family RUN_r2 contains model_r2 and post_proc_r1, etc.
\end{lstlisting}

As described in Section~\ref{FamilyTriggers} family names can be used to
trigger all members at once:
\begin{lstlisting}
    graph = "foo => FAMILY"
\end{lstlisting}
or to trigger off all members:
\begin{lstlisting}
    graph = "FAMILY:succeed-all => bar"
\end{lstlisting}
or to trigger off any members:
\begin{lstlisting}
    graph = "FAMILY:succeed-any => bar"
\end{lstlisting}

If the members of \lstinline=FAMILY= were generated with parameters, you can
also trigger them all at once with parameter notation:
\begin{lstlisting}
    graph = "foo => member<m>"
\end{lstlisting}
Similarly, to trigger off all members:
\begin{lstlisting}
    graph = "member<m> => bar"
    # (member<m>:fail etc., for other trigger types)
\end{lstlisting}

Family names are still needed in the graph, however, to succinctly express
``succeed-any'' triggering semantics, and all-to-all or any-to-all triggering:
\begin{lstlisting}
    graph = "FAM1:succeed-any => FAM2"
\end{lstlisting}
(Direct all-to-all and any-to-all family triggering is not recommended for
efficiency reasons though - see Section~\ref{EfficientInterFamilyTriggering}).

For family {\em member-to-member} triggering use parameterized members.
For example, if family \lstinline=OBS_GET= has members \lstinline=get<obs>= and
family \lstinline=OBS_PROC= has members \lstinline=proc<obs>= then this graph:
\begin{lstlisting}
    graph = "get<obs> => proc<obs>"  # for obs = ship, buoy, plane
\end{lstlisting}
expands to:
\begin{lstlisting}
    get_ship => proc_ship
    get_buoy => proc_buoy
    get_plane => proc_plane
\end{lstlisting}

\subsubsection{Parameterized Cycling}
\label{Parameterized Cycling}

Two ways of constructing cycling systems are described and contrasted in
Section~\ref{Workflows For Cycling Systems}.  For most purposes use of a proper
{\em cycling workflow} is recommended, wherein cylc incrementally generates the
date-time sequence and extends the workflow, potentially indefinitely, at run
time. For smaller systems of finite duration, however, parameter expansion
can be used to generate a sequence of pre-defined tasks as a proxy for cycling.

Here's a cycling workflow of two-monthly model runs for one year,
with previous-instance model dependence (e.g.\ for model restart files):
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020-01
    final cycle point = 2020-12
    [[dependencies]]
        [[[R1]]]  # Run once, at the initial point.
            graph = "prep => model"
        [[[P2M]]]  # Run at 2-month intervals between the initial and final points.
            graph = "model[-P2M] => model => post_proc & archive"
[runtime]
    [[model]]
        script = "run-model $CYLC_TASK_CYCLE_POINT"
\end{lstlisting}

And here's how to do the same thing with parameterized tasks:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    [[parameters]]
        chunk = 1..6
[scheduling]
    [[dependencies]]
        graph = """prep => model<chunk=1>
                     model<chunk-1> => model<chunk> =>
                       post_proc<chunk> & archive<chunk>"""
[runtime]
    [[model<chunk>]]
        script = """
# Compute start date from chunk index and interval, then run the model.
INITIAL_POINT=2020-01
INTERVAL_MONTHS=2
OFFSET_MONTHS=(( (CYLC_TASK_PARAM_chunk - 1)*INTERVAL_MONTHS ))
OFFSET=P${OFFSET_MONTHS}M  # e.g. P4M for chunk=3
run-model $(cylc cyclepoint --offset=$OFFSET $INITIAL_POINT)"""
\end{lstlisting}

The two workflows are shown together in Figure~\ref{fig-eg2}. They both achieve
the same result, and both can include special tasks at the start, end, or
anywhere in between. But as noted earlier the parameterized version has
several disadvantages: it must be finite in extent and not too large; the
date-time arithmetic has to be done by the user; and the full extent of the
workflow will be visible at all times as the suite runs.

\begin{figure}
    \begin{center}
        \includegraphics[width=16cm]{graphics/png/orig/eg2-static.png}
    \end{center}
    \begin{center}
        \includegraphics[width=10cm]{graphics/png/orig/eg2-dynamic.png}
    \end{center}
    \caption[Parameterized (top) and cycling (bottom) versions of the same
    workflow.]{\scriptsize parameterized and cycling versions of the same
      workflow. The first three cycle points are shown in the
      cycling case. The parameterized case does not have ``cycle
    points''.}
    \label{fig-eg2}
\end{figure}

Here's a yearly-cycling suite with four parameterized chunks in each cycle
point:
\begin{lstlisting}
[cylc]
    [[parameters]]
        chunk = 1..4
[scheduling]
    initial cycle point = 2020-01
    [[dependencies]]
        [[[P1Y]]]
            graph = """model<chunk-1> => model<chunk>
                    model<chunk=4>[-P1Y] => model<chunk=1>"""
\end{lstlisting}
Note the inter-cycle trigger that connects the first chunk in each cycle point
to the last chunk in the previous cycle point. Of course it would be simpler
to just use 3-monthly cycling:
\begin{lstlisting}
[scheduling]
    initial cycle point = 2020-01
    [[dependencies]]
        [[[P3M]]]
            graph = "model[-P3M] => model"
\end{lstlisting}

Here's a possible valid use-case for mixed cycling: consider a portable
date-time cycling workflow of model jobs that can each take too long to run on
some supported platforms. This could be handled without changing the cycling
structure of the suite by splitting the run (at each cycle point) into a
variable number of shorter steps, using more steps on less powerful hosts.

\paragraph{Cycle Point And Parameter Offsets At Start-Up}

In cycling workflows cylc ignores anything earlier than the suite initial
cycle point. So this graph:
\begin{lstlisting}
    graph = "model[-P1D] => model"
\end{lstlisting}
simplifies at the initial cycle point to this:
\begin{lstlisting}
    graph = "model"
\end{lstlisting}

Similarly, parameter offsets are ignored if they extend beyond the start of the
parameter value list. So this graph:
\begin{lstlisting}
    graph = "model<chunk-1> => model<chunk>"
\end{lstlisting}
simplifies for \lstinline@chunk=1@ to this:
\begin{lstlisting}
    graph = "model_chunk1"
\end{lstlisting}

Note however that the initial cut-off applies to every parameter list, but only
to cycle point sequences that start at the suite initial cycle point. Therefore
it may be somewhat easier to use parameterized cycling if you need multiple
date-time sequences {\em with different start points} in the same suite. We
plan to allow this sequence-start simplification for any date-time sequence in
the future, not just at the suite initial point, but it needs to be optional
because delayed-start cycling tasks sometimes need to trigger off earlier
cycling tasks.

\subsection{Jinja2}
\label{Jinja2}

{\em This section needs to be revised - the Parameterized Task feature
    introduced in cylc-6.11.0 (see Section~\ref{Parameterized Tasks}) provides
    a cleaner way to auto-generate tasks without coding messy Jinja2 loops.}

Cylc has built in support for the Jinja2 template processor in suite
definitions. Jinja2 variables, mathematical expressions, loop control
structures, conditional logic, etc., are automatically processed to
generate the final suite definition seen by cylc.

The need for Jinja2 processing must be declared with a hash-bang
comment as the first line of the suite.rc file:
\begin{lstlisting}
#!jinja2
# ...
\end{lstlisting}

Potential uses for this include automatic generation of repeated groups
of similar tasks and dependencies, and inclusion or exclusion of entire
suite sections according to the value of a single flag. Consider a
large complicated operational suite and several related parallel test
suites with slightly different task content and structure (the parallel
suites, for instance, might take certain large input files from the
operation or the archive rather than downloading them again) - these can
now be maintained as a single master suite definition that reconfigures
itself according to the value of a flag variable indicating the intended use.

Template processing is the first thing done on parsing a suite
definition so Jinja2 expressions can appear anywhere in the file (inside
strings and namespace headings, for example).

Jinja2 is well documented at \url{http://jinja.pocoo.org/docs}, so here
we just provide an example suite that uses it. The meaning of the
embedded Jinja2 code should be reasonably self-evident to anyone familiar
with standard programming techniques.

\begin{figure}
    \begin{center}
        \includegraphics[width=10cm]{graphics/png/orig/jinja2-ensemble-graph.png}
    \end{center}
    \caption[The Jinja2 ensemble example suite graph.]{\scriptsize
    The Jinja2 ensemble example suite graph.}
    \label{fig-jinja2-ensemble}
\end{figure}

The \lstinline=jinja2.ensemble= example, graphed in
Figure~\ref{fig-jinja2-ensemble}, shows an ensemble of similar tasks
generated using Jinja2:
\lstset{language=suiterc}
\begin{lstlisting}
#!jinja2
{% set N_MEMBERS = 5 %}
[scheduling]
    [[dependencies]]
        graph = """{# generate ensemble dependencies #}
            {% for I in range( 0, N_MEMBERS ) %}
               foo => mem_{{ I }} => post_{{ I }} => bar
            {% endfor %}"""
\end{lstlisting}
Here is the generated suite definition, after Jinja2 processing:
\lstset{language=suiterc}
\begin{lstlisting}
#!jinja2
[scheduling]
    [[dependencies]]
        graph = """
          foo => mem_0 => post_0 => bar
          foo => mem_1 => post_1 => bar
          foo => mem_2 => post_2 => bar
          foo => mem_3 => post_3 => bar
          foo => mem_4 => post_4 => bar
                """
\end{lstlisting}

And finally, the \lstinline=jinja2.cities= example uses variables,
includes or excludes special cleanup tasks according to the value of a
logical flag, and it automatically generates all dependencies and family
relationships for a group of tasks that is repeated for each city in the
suite. To add a new city and associated tasks and dependencies simply
add the city name to list at the top of the file. The suite is graphed,
with the New York City task family expanded, in
Figure~\ref{fig-jinja2-cities}.

\lstset{language=suiterc}
\lstinputlisting{../../../examples/jinja2/cities/suite.rc}
\lstset{language=transcript}

\begin{figure}
    \begin{center}
        \includegraphics[width=16cm]{graphics/png/orig/jinja2-suite-graph.png}
    \end{center}
    \caption[Jinja2 cities example suite graph.]{\scriptsize
    The Jinja2 cities example suite graph, with the
    New York City task family expanded.}
    \label{fig-jinja2-cities}
\end{figure}

\subsubsection{Accessing Environment Variables With Jinja2}

This functionality is not provided by Jinja2 by default, but cylc
automatically imports the user environment to the template in a
dictionary structure called {\em environ}. A usage example:
\begin{lstlisting}
#!Jinja2
#...
[runtime]
    [[root]]
        [[[environment]]]
            SUITE_OWNER_HOME_DIR_ON_SUITE_HOST = {{environ['HOME']}}
\end{lstlisting}
This example is emphasizes that {\em the environment is read on the suite
host at the time the suite definition is parsed} - it is not, for
instance, read at task run time on the task host.

\subsubsection{Custom Jinja2 Filters}
\label{CustomJinja2Filters}

Jinja2 variable values can be modified by ``filters'', using pipe
notation. For example, the built-in \lstinline=trim= filter strips
leading and trailing white space from a string:
\lstset{language=suiterc}
\begin{lstlisting}
{% set MyString = "   dog   " %}
{{ MyString | trim() }}  # "dog"
\end{lstlisting}
(See official Jinja2 documentation for available built-in filters.)

Cylc also supports custom Jinja2 filters. A custom filter is a
single Python function in a source file with the same name as the
function (plus ``.py'' extension) and stored in one of the following
locations:
\begin{myitemize}
    \item \lstinline=$CYLC_DIR/lib/Jinja2Filters/=
    \item \lstinline=[suite definition directory]/Jinja2Filters/=
    \item \lstinline=$HOME/.cylc/Jinja2Filters/=
\end{myitemize}

In the filter function argument list, the first argument is the variable
value to be ``filtered'', and subsequent arguments can be whatever is
needed. Currently there are two custom filters:


\paragraph{pad}

The ``pad'' filter is for padding string values to some
constant length with a fill character - useful for generating task names
and related values in ensemble suites:

\lstset{language=suiterc}
\begin{lstlisting}
{% for i in range(0,100) %}  # 0, 1, ..., 99
    {% set j = i | pad(2,'0') %}
    A_{{j}}          # A_00, A_01, ..., A_99
{% endfor %}
\end{lstlisting}

\paragraph{strftime}

The ``strftime'' filter can be used to format ISO8601 date-time strings using
an strftime string.

\lstset{language=suiterc}
\begin{lstlisting}
{% set START_CYCLE = '10661004T08+01' %}
{{ START_CYCLE | strftime('%H') }}  # 00
\end{lstlisting}

Examples:

\begin{myitemize}
    \item \lstinline={{START_CYCLE | strftime('%Y')}}= - 1066
    \item \lstinline={{START_CYCLE | strftime('%m')}}= - 10
    \item \lstinline={{START_CYCLE | strftime('%d')}}= - 14
    \item \lstinline={{START_CYCLE | strftime('%H:%M:%S %z')}}= - 08:00:00 +01
\end{myitemize}

It is also possible to parse non-standard date-time strings by passing a
strptime string as the second argument.

Examples:

\begin{myitemize}
    \item \lstinline={{'12,30,2000' | strftime('%m', '%m,%d,%Y')}}= - 12
    \item \lstinline={{'1066/10/14 08:00:00' | strftime('%Y%m%dT%H', '%Y/%m/%d %H:%M:%S')}}= - 10661014T08
\end{myitemize}

\subsubsection{Associative Arrays In Jinja2}

Associative arrays ({\em dicts} in Python) can be very useful.
Here's an example, from \\*
\lstinline=$CYLC_DIR/examples/jinja2/dict=:

\lstset{language=suiterc}
\begin{lstlisting}
#!Jinja2
{% set obs_types = ['airs', 'iasi'] %}
{% set resource = { 'airs':'ncpus=9', 'iasi':'ncpus=20' } %}

[scheduling]
    [[dependencies]]
        graph = OBS
[runtime]
    [[OBS]]
        [[[job]]]
            batch system = pbs
    {% for i in obs_types %}
    [[ {{i}} ]]
        inherit = OBS
        [[[directives]]]
             -I = {{ resource[i] }}
     {% endfor %}
 \end{lstlisting}

Here's the result:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc get-suite-config -i [runtime][airs]directives SUITE
-I = ncpus=9
\end{lstlisting}

\subsubsection{Jinja2 Default Values And Template Inputs}

The values of Jinja2 variables can be passed in from the cylc command
line rather than hardwired in the suite definition.
Here's an example, from \\*
\lstinline=$CYLC_DIR/examples/jinja2/defaults=:

\lstset{language=suiterc}
\begin{lstlisting}
#!Jinja2

[meta]

    title = "Jinja2 example: use of defaults and external input"

    description = """
The template variable FIRST_TASK must be given on the cylc command line
using --set or --set-file=FILE; two other variables, LAST_TASK and
N_MEMBERS can be set similarly, but if not they have default values."""

{% set LAST_TASK = LAST_TASK | default( 'baz' ) %}
{% set N_MEMBERS = N_MEMBERS | default( 3 ) | int %}

{# input of FIRST_TASK is required - no default #}

[scheduling]
    initial cycle point = 20100808T00
    final cycle point   = 20100816T00
    [[dependencies]]
        [[[0]]]
            graph = """{{ FIRST_TASK }} => ENS
                 ENS:succeed-all => {{ LAST_TASK }}"""
[runtime]
    [[ENS]]
{% for I in range( 0, N_MEMBERS ) %}
    [[ mem_{{ I }} ]]
        inherit = ENS
{% endfor %}
\end{lstlisting}

Here's the result:

\lstset{language=transcript}
\begin{lstlisting}
$ cylc list SUITE
Jinja2 Template Error
'FIRST_TASK' is undefined
cylc-list foo  failed:  1

$ cylc list --set FIRST_TASK=bob foo
bob
baz
mem_2
mem_1
mem_0

$ cylc list --set FIRST_TASK=bob --set LAST_TASK=alice foo
bob
alice
mem_2
mem_1
mem_0

$ cylc list --set FIRST_TASK=bob --set N_MEMBERS=10 foo
mem_9
mem_8
mem_7
mem_6
mem_5
mem_4
mem_3
mem_2
mem_1
mem_0
baz
bob
\end{lstlisting}

\lstset{language=suiterc}
Note also that
\lstinline@cylc view --set FIRST_TASK=bob --jinja2 SUITE@ will show the
suite with the Jinja2 variables as set.

{\em Note:} suites started with template variables set on the command
line will {\em restart} with the same settings. However, you can set
them again on the \lstinline=cylc restart= command line if they need to
be overridden.

\subsubsection{Jinja2 Variable Scope}

Jinja2 variable scoping rules may be surprising. Variables set inside a
{\em for loop} block, for instance, are not accessible outside of the block,
so the following will print \lstinline=# FOO is 0=, not \lstinline=# FOO is 9=:

\lstset{language=suiterc}
\begin{lstlisting}
{% set FOO = false %}
{% for item in items %}
    {% if item.check_something() %}
        {% set FOO = true %}
    {% endif %}
{% endfor %}
# FOO is {{FOO}}
\end{lstlisting}

Jinja2 documentation suggests using alternative constructs like the loop else
block or the special \lstinline=loop= variable. More complex use cases can be
handled using \lstinline=namespace= objects which allow propagating of changes
across scopes:

\lstset{language=suiterc}
\begin{lstlisting}
{% set ns = namespace(foo=false) %}
{% for item in items %}
    {% if item.check_something() %}
        {% set ns.foo = true %}
    {% endif %}
{% endfor %}
# FOO is {{ns.foo}}
\end{lstlisting}

For detail, see:
\href{http://jinja.pocoo.org/docs/2.10/templates/#assignments}{Jinja2 Template Designer Documentation > Assignments}

\subsubsection{Raising Exceptions}

Cylc provides two functions for raising exceptions using Jinja2. These
exceptions are raised when the suite.rc file is loaded and will prevent a suite
from running.

Note: These functions must be contained within \lstinline={{= Jinja2
blocks as opposed to \lstinline={%= blocks.

\paragraph{Raise}

The ``raise'' function will result in an error containing the provided text.

\lstset{language=suiterc}
\begin{lstlisting}
{% if not VARIABLE is defined %}
    {{ raise('VARIABLE must be defined for this suite.') }}
{% endif %}
\end{lstlisting}

\paragraph{Assert}

The ``assert'' function will raise an exception containing the text provided in
the second argument providing that the first argument evaluates as False. The
following example is equivalent to the ``raise'' example above.

\lstset{language=suiterc}
\begin{lstlisting}
{{ assert(VARIABLE is defined, 'VARIABLE must be defined for this suite.') }}
\end{lstlisting}

\subsection{Omitting Tasks At Runtime}

It is sometimes convenient to omit certain tasks from the suite at
runtime without actually deleting their definitions from the suite.

Defining [runtime] properties for tasks that do not appear in the suite
graph results in verbose-mode validation warnings that the tasks are
disabled. They cannot be used because the suite graph is what defines
their dependencies and valid cycle points. Nevertheless, it is legal to
leave these orphaned runtime sections in the suite definition because it
allows you to temporarily remove tasks from the suite by simply
commenting them out of the graph.

To omit a task from the suite at runtime but still leave it fully
defined and available for use (by insertion or \lstinline=cylc submit=)
use one or both of [scheduling][[special task]] lists, {\em include at
start-up} or {\em exclude at start-up} (documented in~\ref{IASU}
and~\ref{EASU}). Then the graph still defines the
validity of the tasks and their dependencies, but they are not actually
loaded into the suite at start-up. Other tasks that depend on the
omitted ones, if any, will have to wait on their insertion at a later
time or otherwise be triggered manually.

Finally, with Jinja2 (\ref{Jinja2}) you can radically alter
suite structure by including or excluding tasks from the [scheduling]
and [runtime] sections according to the value of a single logical flag
defined at the top of the suite.
\subsection{Naked Dummy Tasks And Strict Validation}

A {\em naked dummy task} appears in the suite graph but has no
explicit runtime configuration section. Such tasks automatically
inherit the default ``dummy task'' configuration from the root
namespace. This is very useful because it allows functional suites to
be mocked up quickly for test and demonstration purposes by simply
defining the graph. It is somewhat dangerous, however, because there
is no way to distinguish an intentional naked dummy task from one
generated by typographic error: misspelling a task name in the graph
results in a new naked dummy task replacing the intended task in the
affected trigger expression; and misspelling a task name in a runtime
section heading results in the intended task becoming a dummy task
itself (by divorcing it from its intended runtime config section).

To avoid this problem any dummy task used in a real suite should not be
naked - i.e.\ it should have an explicit entry in under the runtime
section of the suite definition, even if the section is empty. This
results in exactly the same dummy task behaviour, via implicit
inheritance from root, but it allows use of
\lstinline=cylc validate --strict=
to catch errors in task names by failing the suite if any naked dummy
tasks are detected.

\section{Task Implementation}
\label{TaskImplementation}

Existing scripts and executables can be used as cylc tasks without modification
so long as they return standard exit status - zero on success, non-zero
for failure - and do not spawn detaching processes internally
(see~\ref{DetachingJobs}).

\subsection{Task Job Scripts}
\label{JobScripts}

When the suite dameon determines that a task is ready to run it generates a
{\em job script} that embodies the task runtime configuration in the suite.rc
file, and submits it to the configured job host and batch system
(see~\ref{TaskJobSubmission}).

Task job scripts are written to the suite's job log directory. They can be
printed with \lstinline=cylc cat-log= or generated and printed with
\lstinline=cylc jobscript=.

\subsection{Inlined Tasks}

Task {\em script} items can be multi-line strings of \lstinline=bash=  code, so
many tasks can be entirely inlined in the suite.rc file. For anything more than
a few lines of code, however, we recommend using external shell scripts to allow
independent testing, re-use, and shell mode editing.

\subsection{Task Messages}

Tasks messages can be sent back to the suite server program to report completed
outputs and arbitrary messages of different severity levels.

Some types of message - in addition to events like task failure -  can
optionally trigger execution of event handlers in the suite server program
(see~\ref{EventHandling}).

Normal severity messages are printed to \lstinline=job.out= and logged by the
suite server program:
\lstset{language=bash}
\begin{lstlisting}
cylc message "Hello from ${CYLC_TASK_ID}"
\end{lstlisting}

CUSTOM severity messages are printed to \lstinline=job.out=, logged by the
suite server program, and can be used to trigger {\em custom} event handlers:
\lstset{language=bash}
\begin{lstlisting}
cylc message -p CUSTOM "data available for ${CYLC_TASK_CYCLE_POINT}"
\end{lstlisting}
Task output messages, used for triggering other tasks, can also be sent with
custom severity if need be

WARNING severity messages are printed to \lstinline=job.err=, logged by the
suite server program, and can be passed to {\em warning} event handlers:
\begin{lstlisting}
cylc message -p WARNING "Uh-oh, something's not right here."
\end{lstlisting}

CRITICAL severity messages are printed to \lstinline=job.err=, logged by the
suite server program, and can be passed to {\em critical} event handlers:
\begin{lstlisting}
cylc message -p CRITICAL "ERROR occurred in process X!"
\end{lstlisting}

\subsection{Aborting Job Scripts on Error}

Task job scripts use \lstinline=set -x= to abort on any error, and
trap ERR, EXIT, and SIGTERM to send task failed messages back to the
suite server program before aborting. Other scripts called from job scripts
should therefore abort with standard non-zero exit status on error, to trigger
the job script error trap.

To prevent a command that is expected to generate a non-zero exit status from
triggering the exit trap, protect it with a control statement such as:
\lstset{language=bash}
\begin{lstlisting}
if cmp FILE1 FILE2; then
    :  # success: do stuff
else
    :  # failure: do other stuff
fi
\end{lstlisting}

Task job scripts also use \lstinline=set -u= to abort on referencing any
undefined variable (useful for picking up typos); and \lstinline=set -o pipefail=
to abort if any part of a pipe fails (by default the shell only returns the
exit status of the final command in a pipeline).

\subsubsection{Custom Failure Messages}

Critical events normally warrant aborting a job script rather than just sending
a message. As described just above, \lstinline=exit 1= or any failing command
not protected by the surrounding scripting will cause a job script to abort and
report failure to the suite server program, potentially triggering a
{\em failed} task event handler.

For failures detected by the scripting you could send a critical message back
before aborting, potentially triggering a {\em critical} task event handler:
\begin{lstlisting}
if ! /bin/false; then
  cylc message -p CRITICAL "ERROR: /bin/false failed!"
  exit 1
fi
\end{lstlisting}

To abort a job script with a custom message that can be passed to a {\em
failed} task event handler, use the built-in \lstinline=cylc__job_abort= shell
function:
\begin{lstlisting}
if ! /bin/false; then
  cylc__job_abort "ERROR: /bin/false failed!"
fi
\end{lstlisting}

\subsection{Avoid Detaching Processes}
\label{DetachingJobs}

\lstset{language=transcript}
If a task script starts background sub-processes and does not wait on them, or
internally submits jobs to a batch scheduler and then exits immediately, the
detached processes will not be visible to cylc and the task will appear to
finish when the top-level script finishes. You will need to modify scripts
like this to make them execute all sub-processes in the foreground (or use the
shell \lstinline=wait= command to wait on them before exiting) and to prevent
job submission commands from returning before the job completes (e.g.\
\lstinline=llsubmit -s= for Loadleveler,
\lstinline=qsub -sync yes= for Sun Grid Engine, and
\lstinline@qsub -W block=true@ for PBS).

If this is not possible - perhaps you don't have control over the script
or can't work out how to fix it - one alternative approach is to use another
task to repeatedly poll for the results of the detached processes:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "model => checker => post-proc"
[runtime]
    [[model]]
        # Uh-oh, this script does an internal job submission to run model.exe:
        script = "run-model.sh"
    [[checker]]
        # Fail and retry every minute (for 10 tries at the most) if model's
        # job.done indicator file does not exist yet.
        script = "[[ ! -f $RUN_DIR/job.done ]] && exit 1"
        [[[job]]]
            execution retry delays = 10 * PT1M
\end{lstlisting}

\section{Task Job Submission and Management}
\label{TaskJobSubmission}

For the requirements a command, script, or program, must fulfill in order
to function as a cylc task, see~\ref{TaskImplementation}.
This section explains how tasks are submitted by the suite server program when
they are ready to run, and how to define new batch system handlers.

When a task is ready cylc generates a job script (see~\ref{JobScripts}). The
job script is submitted to run by the {\em batch system} chosen for
the task. Different tasks can use different batch systems.  Like
other runtime properties, you can set a suite default batch system and
override it for specific tasks or families:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
   [[root]] # suite defaults
        [[[job]]]
            batch system = loadleveler
   [[foo]] # just task foo
        [[[job]]]
            batch system = at
\end{lstlisting}

\subsection{Supported Job Submission Methods}
\label{AvailableMethods}

Cylc supports a number of commonly used batch systems.
See~\ref{CustomJobSubmissionMethods} for how to add new job
submission methods.

\subsubsection{background}

Runs task job scripts as Unix background processes.

If an execution time limit is specified for a task, its job will be wrapped
by the \lstinline=timeout= command.

\subsubsection{at}

Submits task job scripts to the rudimentary Unix \lstinline=at= scheduler. The
\lstinline=atd= daemon must be running.

If an execution time limit is specified for a task, its job will be wrapped
by the \lstinline=timeout= command.

\subsubsection{loadleveler}

Submits task job scripts to loadleveler by the \lstinline=llsubmit= command.
Loadleveler directives can be provided in the suite.rc file:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = loadleveler
            execution time limit = PT10M
        [[[directives]]]
            foo = bar
            baz = qux
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
# @ foo = bar
# @ baz = qux
# @ wall_clock_limit = 660,600
# @ queue
\end{lstlisting}

If restart=yes is specified as a directive for loadleveler, the job will
automatically trap SIGUSR1, which loadleveler may use to preempt the job. On
trapping SIGUSR1, the job will inform the suite that it has been vacated by
loadleveler. This will put it back to the submitted state, until it starts
running again.

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=wall_clock_limit= directive. The setting is assumed to be the soft
limit. The hard limit will be set by adding an extra minute to the soft limit.
Do not specify the \lstinline=wall_clock_limit= directive explicitly if
\lstinline=execution time limit= is specified. Otherwise, the execution time
limit known by the suite may be out of sync with what is submitted to the batch
system.

\subsubsection{lsf}

Submits task job scripts to IBM Platform LSF by the \lstinline=bsub= command.
LSF directives can be provided in the suite.rc file:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = lsf
            execution time limit = PT10M
        [[[directives]]]
            -q = foo
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
#BSUB -q = foo
#BSUB -W = 10
\end{lstlisting}

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=-W= directive. Do not specify the \lstinline=-W= directive
explicitly if \lstinline=execution time limit= is specified. Otherwise, the
execution time limit known by the suite may be out of sync with what is
submitted to the batch system.

\subsubsection{pbs}

Submits task job scripts to PBS (or Torque) by the \lstinline=qsub= command.
PBS directives can be provided in the suite.rc file:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = pbs
            execution time limit = PT1M
        [[[directives]]]
            -V =
            -q = foo
            -l nodes = 1
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
#PBS -V
#PBS -q foo
#PBS -l nodes=1
#PBS -l walltime=60
\end{lstlisting}

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=-l walltime= directive. Do not specify the \lstinline=-l walltime=
directive explicitly if \lstinline=execution time limit= is specified.
Otherwise, the execution time limit known by the suite may be out of sync with
what is submitted to the batch system.

\subsubsection{moab}

Submits task job scripts to the Moab workload manager by the \lstinline=msub=
command.  Moab directives can be provided in the suite.rc file; the syntax is
very similar to PBS:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = moab
            execution time limit = PT1M
        [[[directives]]]
            -V =
            -q = foo
            -l nodes = 1
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
#PBS -V
#PBS -q foo
#PBS -l nodes=1
#PBS -l walltime=60
\end{lstlisting}
(Moab understands \lstinline=#PBS= directives).

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=-l walltime= directive. Do not specify the \lstinline=-l walltime=
directive explicitly if \lstinline=execution time limit= is specified.
Otherwise, the execution time limit known by the suite may be out of sync with
what is submitted to the batch system.

\subsubsection{sge}

Submits task job scripts to Sun/Oracle Grid Engine by the \lstinline=qsub=
command.  SGE directives can be provided in the suite.rc file:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = sge
            execution time limit = P1D
        [[[directives]]]
            -cwd =
            -q = foo
            -l h_data = 1024M
            -l h_rt = 24:00:00
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
# DIRECTIVES
#$ -cwd
#$ -q foo
#$ -l h_data=1024M
#$ -l h_rt=24:00:00
\end{lstlisting}

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=-l h_rt= directive. Do not specify the \lstinline=-l h_rt=
directive explicitly if \lstinline=execution time limit= is specified.
Otherwise, the execution time limit known by the suite may be out of sync with
what is submitted to the batch system.

\subsubsection{slurm}

Submits task job scripts to Simple Linux Utility for Resource Management by the
\lstinline=sbatch= command. SLURM directives can be provided in the suite.rc
file (note that since not all SLURM commands have a short form, cylc requires
the long form directives):
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[my_task]]
        [[[job]]]
            batch system = slurm
            execution time limit = PT1H
        [[[directives]]]
            --nodes = 5
            --account = QXZ5W2
\end{lstlisting}
These are written to the top of the task job script like this:
\lstset{language=bash}
\begin{lstlisting}
#!/bin/bash
#SBATCH --nodes=5
#SBATCH --time=60:00
#SBATCH --account=QXZ5W2
\end{lstlisting}

If \lstinline=execution time limit= is specified, it is used to generate the
\lstinline=--time= directive. Do not specify the \lstinline=--time=
directive explicitly if \lstinline=execution time limit= is specified.
Otherwise, the execution time limit known by the suite may be out of sync with
what is submitted to the batch system.

\subsubsection{Default Directives Provided}

For batch systems that use job file directives (PBS, Loadleveler,
etc.) default directives are provided to set the job name, stdout and stderr
file paths, and the execution time limit (if specified).

Cylc constructs the job name string using a combination of the task ID and the
suite name. PBS fails a job submit if the job name in \lstinline=-N name= is
too long. For version 12 or below, this is 15 characters. For version 13, this
is 236 characters. The default setting will truncate the job name string to 15
characters. If you have PBS 13 at your site, you should modify your site's
global configuration file to allow the job name to be longer. (See also
Section~\ref{JobNameLengthMaximum}.) For example:

\begin{lstlisting}
[hosts]
    [[myhpc*]]
        [[[batch systems]]]
            [[[[pbs]]]]
                # PBS 13
                job name length maximum = 236
\end{lstlisting}

\subsubsection{Directives Section Quirks (PBS, SGE, ...) }

To specify an option with no argument, such as \lstinline=-V= in PBS or
\lstinline=-cwd= in SGE you must give a null string as the directive value in
the suite.rc file.

The left hand side of a setting (i.e.\ the string before the first equal sign)
must be unique. To specify multiple values using an option such as
\lstinline=-l= option in PBS, SGE, etc., either specify all items in a single
line:

\begin{lstlisting}
-l=select=28:ncpus=36:mpiprocs=18:ompthreads=2:walltime=12:00:00
\end{lstlisting}

(Left hand side is \lstinline=-l=. A second \lstinline@-l=...@ line will
override the first.)

Or separate the items (note: no equal sign after \lstinline=-l=):

\begin{lstlisting}
-l select=28
-l ncpus=36
-l mpiprocs=18
-l ompthreads=2
-l walltime=12:00:00
\end{lstlisting}

(Left hand sides are now \lstinline=-l select=, \lstinline=-l ncpus=, etc.)

\subsection{Task stdout And stderr Logs}
\label{WhitherStdoutAndStderr}

When a task is ready to run cylc generates a filename root to be used
for the task job script and log files. The filename containing the task
name, cycle point, and a submit number that increments if the same task is
re-triggered multiple times:

\lstset{language=bash}
\begin{lstlisting}
# task job script:
~/cylc-run/tut/oneoff/basic/log/job/1/hello/01/job
# task stdout:
~/cylc-run/tut/oneoff/basic/log/job/1/hello/01/job.out
# task stderr:
~/cylc-run/tut/oneoff/basic/log/job/1/hello/01/job.err
\end{lstlisting}

How the stdout and stderr streams are directed into these files depends
on the batch system. The \lstinline=background= method just uses
appropriate output redirection on the command line, as shown above. The
\lstinline=loadleveler= method writes appropriate directives to the job
script that is submitted to loadleveler.

Cylc obviously has no control over the stdout and stderr output from
tasks that do their own internal output management (e.g.\ tasks
that submit internal jobs and direct the associated output to other
files). For less internally complex tasks, however, the files referred
to here will be complete task job logs.

Some batch systems, such as \lstinline=pbs=, redirect a job's stdout
and stderr streams to a separate cache area while the job is running. The
contents are only copied to the normal locations when the job completes. This
means that \lstinline=cylc cat-log= or the gcylc GUI will be unable to find the
job's stdout and stderr streams while the job is running. Some sites with these
batch systems are known to provide commands for viewing and/or
tail-follow a job's stdout and stderr streams that are redirected to these
cache areas. If this is the case at your site, you can configure cylc to make
use of the provided commands by adding some settings to the global site/user
config. E.g.:

\begin{lstlisting}
[hosts]
    [[HOST]]  # <= replace this with a real host name
        [[[batch systems]]]
            [[[[pbs]]]]
                err tailer = qcat -f -e \%(job_id)s
                out tailer = qcat -f -o \%(job_id)s
                err viewer = qcat -e \%(job_id)s
                out viewer = qcat -o \%(job_id)s
\end{lstlisting}

\subsection{Overriding The Job Submission Command}
\label{CommandTemplate}

\lstset{language=suiterc}
To change the form of the actual command used to submit a job you do not
need to define a new batch system handler; just override the
\lstinline=command template= in the relevant job submission sections of
your suite.rc file:
\begin{lstlisting}
[runtime]
    [[root]]
        [[[job]]]
            batch system = loadleveler
            # Use '-s' to stop llsubmit returning
            # until all job steps have completed:
            batch submit command template = llsubmit -s %(job)s
\end{lstlisting}
As explained in~\ref{SuiteRCReference}
the template's \%(job)s will be substituted by the job file path.

\subsection{Job Polling}

For supported batch systems, one-way polling can be used to determine actual
job status: the suite server program executes a process on the task host, by
non-interactive ssh, to interrogate the batch queueing system there, and to
read a {\em status file} that is automatically generated by the task job script
as it runs.

Polling may be required to update the suite state correctly after unusual
events such as a machine being rebooted with tasks running on it, or network
problems that prevent task messages from getting back to the suite host.

Tasks can be polled on demand by right-clicking on them in gcylc or using the
\lstinline=cylc poll= command.

Tasks are polled automatically, once, if they timeout while queueing in a
batch scheduler and submission timeout is set. (See~\ref{TaskEventHandling} for
how to configure timeouts).

Tasks are polled multiple times, where necessary, when they exceed their
execution time limits. These are normally set with some initial delays to allow
the batch systems to kill the jobs.
(See~\ref{ExecutionTimeLimitPollingIntervals} for how to configure the polling
intervals).

Any tasks recorded in the {\em submitted} or {\em running} states at suite
restart are automatically polled to determine what happened to them while the
suite was down.

Regular polling can also be configured as a health check on tasks submitted to
hosts that are known to be flaky, or as the sole method of determining task
status on hosts that do not allow task messages to be routed back to the suite
host.

To use polling instead of task-to-suite messaging set
\lstinline@task communication method = poll@
in cylc site and user global config (see~\ref{task_comms_method}).
The default polling intervals can be overridden for all suites there too
(see~\ref{submission_polling} and~\ref{execution_polling}), or in specific
suite definitions (in which case polling will be done regardless of the
task communication method configured for the host;
see~\ref{SubmissionPollingIntervals} and~\ref{ExecutionPollingIntervals}).

Note that regular polling is not as efficient as task messaging in updating
task status, and it should be used sparingly in large suites.

Note that for polling to work correctly, the batch queueing system must have a
job listing command for listing your jobs, and that the job listing must
display job IDs as they are returned by the batch queueing system submit
command. For example, for pbs, moab and sge, the \lstinline=qstat= command
should list jobs with their IDs displayed in exactly the same format as they
are returned by the \lstinline=qsub= command.

\subsection{Job Killing}

For supported batch systems, the suite server program can execute a process on
the task host, by non-interactive ssh, to kill a submitted or running job
according to its batch system.

Tasks can be killed on demand by right-clicking on them in gcylc or using the
\lstinline=cylc kill= command.

\subsection{Execution Time Limit}

You can specify an \lstinline=execution time limit= for all supported job
submission methods. E.g.:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[task-x]]
        [[[job]]]
            execution time limit = PT1H
\end{lstlisting}

For tasks running with \lstinline=background= or \lstinline=at=, their jobs
will be wrapped using the \lstinline=timeout= command. For all other methods,
the relevant time limit directive will be added to their job files.

The \lstinline=execution time limit= setting will also inform the suite when a
a task job should complete by. If a task job has not reported completing within
the specified time, the suite will poll the task job. (The default
setting is PT1M, PT2M, PT7M. The accumulated times for these intervals will be
roughly 1 minute, 1 + 2 = 3 minutes and 1 + 2 + 7 = 10 minutes after a task job
exceeds its execution time limit.)

\subsubsection{Execution Time Limit and Execution Timeout}

If you specify an \lstinline=execution time limit= the
\lstinline=execution timeout event handler= will only be called if the job has
not completed after the final poll (by default, 10 min after the time limit).
This should only happen if the submission method you are using is not enforcing
wallclock limits (unlikely) or you are unable to contact the machine to confirm
the job status.

If you specify an \lstinline=execution timeout= and not an
\lstinline=execution time limit= then the
\lstinline=execution timeout event handler= will be called as soon as the
specified time is reached. The job will also be polled to check its latest
status (possibly resulting in an update in its status and the calling of the
relevant event handler). This behaviour is deprecated, which users should avoid
using.

If you specify an \lstinline=execution timeout= and an
\lstinline=execution time limit= then the execution timeout setting will be
ignored.

\subsection{Custom Job Submission Methods}
\label{CustomJobSubmissionMethods}

Defining a new batch system handler requires a little Python programming. Use
the built-in handlers as examples, and read the documentation in
\lstinline=lib/cylc/batch_sys_manager.py=.

\lstset{language=Python}

\subsubsection{An Example}

The following \lstinline=qsub.py= module overrides the built-in {\em pbs}
batch system handler to to change the directive prefix from \lstinline=#PBS= to
\lstinline=#QSUB=:

\begin{lstlisting}
#!/usr/bin/env python

from cylc.batch_sys_handlers.pbs import PBSHandler

class QSUBHandler(PBSHandler):
    DIRECTIVE_PREFIX = "#QSUB "

BATCH_SYSTEM_HANDLER = QSUBHandler()
\end{lstlisting}

If this is in the Python search path (see~\ref{Where To Put Batch System
Handler Modules} below) you can use it by name in suite definitions:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = "a"
[runtime]
    [[root]]
        [[[job]]]
            batch system = qsub  # <---!
            execution time limit = PT1M
        [[[directives]]]
            -l nodes = 1
            -q = long
            -V =
\end{lstlisting}

Generate a job script to see the resulting directives:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc register test $HOME/test
$ cylc jobscript test a.1 | grep QSUB
#QSUB -e /home/oliverh/cylc-run/my.suite/log/job/1/a/01/job.err
#QSUB -l nodes=1
#QSUB -l walltime=60
#QSUB -o /home/oliverh/cylc-run/my.suite/log/job/1/a/01/job.out
#QSUB -N a.1
#QSUB -q long
#QSUB -V
\end{lstlisting}

(Of course this suite will fail at run time because we only changed the
directive format, and PBS does not accept \lstinline=#QSUB= directives in
reality).

\subsubsection{Where To Put Batch System Handler Modules}
\label{Where To Put Batch System Handler Modules}

{\em Custom batch system handlers must be installed on suite and job
hosts} in one of these locations:
\begin{myitemize}
    \item under \lstinline=SUITE-DEF-PATH/lib/python/=
    \item under \lstinline=CYLC-PATH/lib/cylc/batch_sys_handlers/=
    \item or anywhere in \lstinline=$PYTHONPATH=
\end{myitemize}

(A note for Rose users: \lstinline=rose suite-run= automatically installs
\lstinline=SUITE-DEF-PATH/lib/python/= to job hosts).

\section{Running Suites}
\label{RunningSuites}

This chapter currently features a diverse collection of topics related
to running suites. Please also see the Tutorial (\ref{Tutorial}) and
command documentation (\ref{CommandReference}), and experiment with
plenty of examples.

\subsection{Suite Start-Up}
\label{SuiteStartUp}

There are three ways to start a suite running: {\em cold start} and {\em warm
start}, which start from scratch; and {\em restart}, which starts from a prior
suite state checkpoint. The only difference between cold starts and warm starts
is that warm starts start from a point beyond the suite initial cycle point.

Once a suite is up and running it is typically a restart that is needed most
often (but see also \lstinline=cylc reload=). {\em Be aware that cold and warm
starts wipe out prior suite state, so you can't go back to a restart if you
decide you made a mistake.}

\subsubsection{Cold Start}
\label{Cold Start}

A cold start is the primary way to start a suite run from scratch:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc run SUITE [INITIAL_CYCLE_POINT]
\end{lstlisting}
The initial cycle point may be specified on the command line or in the suite.rc
file. The scheduler starts by loading the first instance of each task at the
suite initial cycle point, or at the next valid point for the task.

\subsubsection{Warm Start}
\label{Warm Start}

A warm start runs a suite from scratch like a cold start, but from the
beginning of a given cycle point that is beyond the suite initial cycle point.
This is generally inferior to a {\em restart} (which loads a previously
recorded suite state - see~\ref{RestartingSuites}) because it may result in
some tasks rerunning. However, a warm start may be required if a restart is not
possible, e.g.\ because the suite run database was accidentally deleted. The
warm start cycle point must be given on the command line:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc run --warm SUITE [START_CYCLE_POINT]
\end{lstlisting}
The original suite initial cycle point is preserved, but all tasks and
dependencies before the given warm start cycle point are ignored.

The scheduler starts by loading a first instance of each task at the warm
start cycle point, or at the next valid point for the task.
\lstinline=R1=-type tasks behave exactly the same as other tasks - if their
cycle point is at or later than the given start cycle point, they will run; if
not, they will be ignored.

\subsubsection{Restart and Suite State Checkpoints}
\label{RestartingSuites}

At restart (see \lstinline=cylc restart --help=) a suite server program
initializes its task pool from a previously recorded checkpoint state. By
default the latest automatic checkpoint - which is updated with every task
state change - is loaded so that the suite can carry on exactly as it was just
before being shut down or killed.

\lstset{language=transcript}
\begin{lstlisting}
$ cylc restart SUITE
\end{lstlisting}

Tasks recorded in the `submitted' or `running' states are automatically polled 
(see Section~\ref{Task Job Polling}) at start-up to determine what happened to
them while the suite was down.

\paragraph{Restart From Latest Checkpoint}

To restart from the latest checkpoint simply invoke the \lstinline=cylc restart=
command with the suite name (or select `restart' in the GUI suite start dialog
window):

\lstset{language=transcript}
\begin{lstlisting}
$ cylc restart SUITE
\end{lstlisting}

\paragraph{Restart From Another Checkpoint}

Suite server programs automatically update the ``latest'' checkpoint every time
a task changes state, and at every suite restart, but you can also take
checkpoints at other times. To tell a suite server program to checkpoint its
current state:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc checkpoint SUITE-NAME CHECKPOINT-NAME
\end{lstlisting}

The 2nd argument is a name to identify the checkpoint later with:
\begin{lstlisting}
$ cylc ls-checkpoints SUITE-NAME
\end{lstlisting}

For example, with checkpoints named `bob', `alice', and `breakfast':
\begin{lstlisting}
$ cylc ls-checkpoints SUITE-NAME
#######################################################################
# CHECKPOINT ID (ID|TIME|EVENT)
1|2017-11-01T15:48:34+13|bob
2|2017-11-01T15:48:47+13|alice
3|2017-11-01T15:49:00+13|breakfast
...
0|2017-11-01T17:29:19+13|latest
\end{lstlisting}

To see the actual task state content of a given checkpoint ID (if you need to),
for the moment you have to interrogate the suite DB, e.g.:

\begin{lstlisting}
$ sqlite3 ~/cylc-run/SUITE-NAME/log/db \
    'select * from task_pool_checkpoints where id == 3;'
3|2012|model|1|running|
3|2013|pre|0|waiting|
3|2013|post|0|waiting|
3|2013|model|0|waiting|
3|2013|upload|0|waiting|
\end{lstlisting}

Note that a checkpoint captures the instantaneous state of every task in the
suite, including any tasks that are currently active, so you may want to be
careful where you do it. Tasks recorded as active are polled automatically on
restart to determine what happened to them. 

The checkpoint ID 0 (zero) is always used for latest state of the suite, which
is updated continuously as the suite progresses. The checkpoint IDs of earlier
states are positive integers starting from 1, incremented each time a new
checkpoint is stored. Currently suites automatically store checkpoints before
and after reloads, and on restarts (using the latest checkpoints before the
restarts).

Once you have identified the right checkpoint, restart the suite like this:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc restart --checkpoint=CHECKPOINT-ID SUITE
\end{lstlisting}
or enter the checkpoint ID in the space provided in the GUI restart window.

\paragraph{Checkpointing With A Task}

Checkpoints can be generated automatically at particular points in the
workflow by coding tasks that run the \lstinline=cylc checkpoint= command:

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
   [[dependencies]]
      [[[PT6H]]]
          graph = "pre => model => post => checkpointer"
[runtime]
   # ...
   [[checkpointer]]
      script = """
wait "${CYLC_TASK_MESSAGE_STARTED_PID}" 2>/dev/null || true
cylc checkpoint ${CYLC_SUITE_NAME} CP-${CYLC_TASK_CYCLE_POINT}
               """
\end{lstlisting}

Note that we need to \lstinline=wait= on the ``task started'' message - which
is sent in the background to avoid holding tasks up in a network outage - to
ensure that the checkpointer task is correctly recorded as running in the
checkpoint (at restart the suite server program will poll to determine that
that task job finished successfully). Otherwise it may be recorded in the
waiting state and, if its upstream dependencies have already been cleaned up,
it will need to be manually reset from waiting to succeeded after the restart
to avoid stalling the suite.

\paragraph{Behaviour of Tasks on Restart}

All tasks are reloaded in exactly their checkpointed states. Failed tasks are
not automatically resubmitted at restart in case the underlying problem has not
been addressed yet.

Tasks recorded in the submitted or running states are automatically polled on
restart, to see if they are still waiting in a batch queue, still running, or
if they succeeded or failed while the suite was down. The suite state will be
updated automatically according to the poll results.

Existing instances of tasks removed from the suite definition before restart
are not removed from the task pool automatically, but they will not spawn new
instances. They can be removed manually if necessary,
with~\lstinline=cylc remove=.

Similarly, instances of new tasks added to the suite definition before
restart are not inserted into the task pool automatically, because it is
very difficult in general to automatically determine the cycle point of
the first instance. Instead, the first instance of a new task should be
inserted manually at the right cycle point, with~\lstinline=cylc insert=.

\subsection{Reloading The Suite Definition At Runtime}

The \lstinline=cylc reload= command tells a suite server program to reload its
suite definition at run time. This is an alternative to shutting a suite down
and restarting it after making changes.

As for a restart, existing instances of tasks removed from the suite definition
before reload are not removed from the task pool automatically, but they
will not spawn new instances. They can be removed manually if necessary,
with~\lstinline=cylc remove=.

Similarly, instances of new tasks added to the suite definition before
reload are not inserted into the pool automatically. The first instance of each
must be inserted manually at the right cycle point, with~\lstinline=cylc insert=.

\subsection{Task Job Access To Cylc}
\label{HowTasksGetAccessToCylc}

Task jobs need access to Cylc on the job host, primarily for task messaging,
but also to allow user-defined task scripting to run other Cylc commands.

Cylc should be installed on job hosts as on suite hosts, with different releases
installed side-by-side and invoked via the central Cylc wrapper according to
the value of \lstinline=$CYLC_VERSION= - see Section~\ref{InstallCylc}. Task
job scripts set \lstinline=$CYLC_VERSION= to the version of the parent suite
server program, so that the right Cylc will be invoked by jobs on the job host.

Access to the Cylc executable (preferably the central wrapper as just
described) for different job hosts can be configured using site and user
global configuration files (on the suite host). If the environment for running
the Cylc executable is only set up correctly in a login shell for a given host,
you can set \lstinline@[hosts][HOST]use login shell = True@ for the relevant
host (this is the default, to cover more sites automatically). If the
environment is already correct without the login shell, but the Cylc executable
is not in \lstinline=$PATH=, then \lstinline=[hosts][HOST]cylc executable= can
be used to specify the direct path to the executable.

To customize the environment more generally for Cylc on jobs hosts,
use of \lstinline=job-init-env.sh= is described in Section~\ref{Configure
Environment on Job Hosts}.

\subsection{The Suite Contact File}
\label{The Suite Contact File}

At start-up, suite server programs write a {\em suite contact file}
\lstinline=$HOME/cylc-run/SUITE/.service/contact= that records suite host,
user, port number, process ID, Cylc version, and other information. Client
commands can read this file, if they have access to it, to find the target
suite server program.

\subsection{Task Job Polling}
\label{Task Job Polling}

At any point after job submission task jobs can be {\em polled} to check that
their true state conforms to what is currently recorded by the suite server
program.  See \lstinline=cylc poll --help= for how to poll one or more tasks
manually, or right-click poll a task or family in GUI.

Polling may be necessary if, for example, a task job gets killed by the
untrappable SIGKILL signal (e.g.\ \lstinline=kill -9 PID=), or if a network
outage prevents task success or failure messages getting through, or if the
suite server program itself is down when tasks finish execution.

To poll a task job the suite server program interrogates the batch system, and the
\lstinline=job.status= file, on the job host. This information is enough to
determine the final task status even if the job finished while the suite
server program was down or unreachable on the network.

\subsubsection{Routine Polling}

Task jobs are automatically polled at certain times: once on job submission
timeout; several times on exceeding the job execution time limit; and at suite
restart any tasks recorded as active in the suite state checkpoint are polled
to find out what happened to them while the suite was down.

Finally, in necessary routine polling can be configured as a way to track job
status on job hosts that do not allow networking routing back to the suite host
for task messaging by HTTPS or ssh.  See~\ref{Polling To Track Job Status}.

\subsection{Tracking Task State}
\label{TaskComms}

Cylc supports three ways of tracking task state on job hosts:
\begin{myitemize}
    \item task-to-suite messaging via HTTPS
    \item task-to-suite messaging via non-interactive ssh to the suite host, then local HTTPS
    \item regular polling by the suite server program
\end{myitemize}

These are explained in the following sections. All three can be used, on
different job hosts, in the same suite if necessary.

If your site prohibits HTTPS and ssh back from job hosts to suite hosts, before resorting
to the polling method you should consider installing dedicated Cylc servers or
VMs inside the HPC trust zone (where HTTPS and ssh should be allowed).

It is also possible to run Cylc suite server programs on HPC login nodes, but this is
not recommended for load, run duration, and GUI reasons.

Finally, it has been suggested that {\em port forwarding} may provide another
solution - but that is beyond the scope of this document.

\subsubsection{HTTPS Task Messaging}

Task job wrappers automatically invoke \lstinline=cylc message= to report
progress back to the suite server program when they begin executing, at normal exit
(success) and abnormal exit (failure).

By default the messaging occurs via an authenticated, HTTPS connection to the
suite server program. This is the preferred task communications method - it is
efficient and direct.

Suite server programs automatically install suite contact information and credentials
on job hosts.  Users only need to do this manually for remote access to
suites on other hosts, or suites owned by other users - see~\ref{RemoteControl}.

\subsubsection{Ssh Task Messaging}

Cylc can be configured to re-invoke task messaging commands on the suite host via
non-interactive ssh (from job host to suite host). Then a local HTTPS
connection is made to the suite server program.

(User-invoked client commands (aside from the GUI, which requires HTTPS) can do
the same thing with the \lstinline=--use-ssh= command option).

This is less efficient than direct HTTPS messaging, but it may be useful at
sites where the HTTPS ports are blocked but non-interactive ssh is allowed.

\subsubsection{Polling to Track Job Status}
\label{Polling To Track Job Status}

Finally, suite server programs can actively poll task jobs at configurable intervals,
via non-interactive ssh to the job host.

Polling is the least efficient task communications method because task state is
updated only at intervals, not when task events actually occur.  However, it
may be needed at sites that do not allow HTTPS or non-interactive ssh from job
host to suite host.

Be careful to avoid spamming task hosts with polling commands. Each poll
opens (and then closes) a new ssh connection.

Polling intervals are configurable under \lstinline=[runtime]= because
they should may depend on the expected execution time. For instance, a
task that typically takes an hour to run might be polled every 10
minutes initially, and then every minute toward the end of its run.
Interval values are used in turn until the last value, which is used
repeatedly
until finished:
\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        [[[job]]]
            # poll every minute in the 'submitted' state:
            submission polling intervals = PT1M
            # poll one minute after foo starts running, then every 10
            # minutes for 50 minutes, then every minute until finished:
            execution polling intervals = PT1M, 5*PT10M, PT1M
\end{lstlisting}
A list of intervals with optional multipliers can be used for both
submission and execution polling, although a single value is probably
sufficient for submission polling. If these items are not configured
default values from site and user global config will be used for the polling
task communication method; polling is not done by default under the
other task communications methods (but it can still be used if you
like).

\subsubsection{Task Communications Configuration}

Here are the default site and user global config items relevant to task state
tracking (see these with \lstinline=cylc get-site-config=):

\lstset{language=suiterc}
\begin{lstlisting}
# SITE AND USER CONFIG

# Task messaging settings affect task-to-suite communications.
[task messaging]
    # If a message send fails, retry after this delay:
    retry interval in seconds = 5
    # If send fails after this many tries, give up trying:
    maximum number of tries = 7

    # This timeout is the same as --comms-timeout for user commands. If
    # set to None (no timeout) messages to non-responsive suites
    # (e.g. suspended with Ctrl-Z) could hang indefinitely.
    connection timeout in seconds = 30

# Setup the communication method details. This is required for
# communications between cylc clients and servers (i.e. between
# suite-connecting commands and guis, and running suite server processes).
[communication]

    # Configure the choice of communication method. Https is configured by
    # default, and this requires the Python OpenSSL package to be installed.
    # Http can be configured to override the default setting.
# SITE ONLY
    method = http

    # Each suite listens on a dedicated network port. The port to bind to is
    # selected randomly from the allowed range of ports.
# SITE ONLY
    base port = 43001

    # This sets the maximum number of suites that can run at once.
# SITE ONLY
    maximum number of ports = 100

[hosts]
    # The default task host is the suite host, i.e. localhost:
    # Add task host sections if local defaults are not sufficient.
    [[HOST]]
       # Method of communication of task progress back to the suite:
        #   1) default - HTTPS via network ports
        #   2) ssh  - re-invoke messaging commands on suite server
        #   3) poll - the suite polls for status of passive tasks
        # HTTPS comms are still required in all cases *on the suite host*
        # for cylc clients (commands etc.) to communicate with suites.
        task communication method = "default" # or "ssh" or "poll"
        # The "poll" method sets a default interval here to ensure no
        # tasks are accidentally left unpolled. You can override this
        # with run-length appropriate intervals under task [runtime]
        # (however this will also result in routine polling under the
        # default or ssh communications).
        default polling interval in minutes = 1.0
\end{lstlisting}

\subsection{The Suite Service Directory}
\label{The Suite Service Directory}

At registration time a {\em suite service directory},
\lstinline=$HOME/cylc-run/<SUITE>/.service/=, is created and populated
with a private passphrase file (containing random text), a self-signed
SSL certificate (see~\ref{ConnectionAuthentication}), and a symlink to the
suite source directory.  An existing passphrase file will not be overwritten
if a suite is re-registered.

At run time, the private suite run database is also written to the service
directory, along with a {\em suite contact file} that records the host,
user, port number, process ID, Cylc version, and other information about the
suite server program. Client commands automatically read daemon targetting
information from the contact file, if they have access to it.

\subsection{File-Reading Commands}

Some Cylc commands and GUI actions parse suite definitions or read other files
from the suite host account, rather than communicate with a suite server
program over the network. In future we plan to have suite server program serve
up these files to clients, but for the moment this functionality requires
read-access to the relevant files on the suite host.

If you are logged into the suite host account, file-reading commands will just
work. 

\subsubsection{Remote Host, Shared Home Directory}

If you are logged into another host with shared home directories (shared
filesystems are common in HPC environments) file-reading commands will just
work because suite files will look ``local'' on both hosts. 

\subsubsection{Remote Host, Different Home Directory}

If you are logged into another host with no shared home directory, file-reading
commands require non-interactive ssh to the suite host account, and use of the
\lstinline=--host= and \lstinline=--user= options to re-invoke the command
on the suite account.

\subsubsection{Same Host, Different User Account}

(This is essentially the same as {\em Remote Host, Different Home Directory}.)


\subsection{Client-Server Interaction}
\label{ConnectionAuthentication}

Cylc server programs listen on dedicated network ports for
HTTPS communications from Cylc clients (task jobs, and user-invoked commands
and GUIs).

Use \lstinline=cylc scan= to see which suites are listening on which ports on
scanned hosts (this lists your own suites by default, but it can show others
too - see \lstinline=cylc scan --help=).

Cylc supports two kinds of access to suite server programs:
\begin{myitemize}
    \item {\em public} (non-authenticated) - the amount of information
      revealed is configurable, see~\ref{PublicAccess}
    \item {\em control} (authenticated) - full control, suite passphrase
      required, see~\ref{passphrases}
\end{myitemize}

\subsubsection{Public Access - No Auth Files}
\label{PublicAccess}

Without a suite passphrase the amount of information revealed by a suite
server program is determined by the public access privilege level set in global
site/user config (\ref{GlobalAuth}) and optionally overidden in suites
(\ref{SuiteAuth}):
\begin{myitemize}
    \item {\em identity} - only suite and owner names revealed
    \item {\em description} - identity plus suite title and description
    \item {\em state-totals} - identity, description, and task state totals
    \item {\em full-read} - full read-only access for monitor and GUI
    \item {\em shutdown} - full read access plus shutdown, but no other
        control.
\end{myitemize}
The default public access level is {\em state-totals}.

The \lstinline=cylc scan= command and the \lstinline=cylc gscan= GUI can print
descriptions and task state totals in addition to basic suite identity, if the
that information is revealed publicly.

\subsubsection{Full Control - With Auth Files}
\label{passphrases}

Suite auth files (passphrase and SSL certificate) give full control. They are
loaded from the suite service directory by the suite server program at
start-up, and used to authenticate subsequent client connections. Passphrases
are used in a secure encrypted challenge-response scheme, never sent in plain
text over the network.

If two users need access to the same suite server program, they must both
possess the passphrase file for that suite. Fine-grained access to a single
suite server program via distinct user accounts is not currently supported.

Suite server programs automatically install their auth and contact files to job
hosts via ssh, to enable task jobs to connect back to the suite server program
for task messaging.

Client programs invoked by the suite owner automatically load the passphrase,
SSL certificate, and contact file too, for automatic connection to suites.

{\em Manual installation of suite auth files is only needed for remote control,
if you do not have a shared filesystem - see below.}


\subsection{GUI-to-Suite Interaction}
\label{GUI-to-Suite Interaction}

The gcylc GUI is mainly a network client to retrieve and display suite status
information from the suite server program, but it can also invoke file-reading
commands to view and graph the suite definition and so on. This is entirely
transparent if the GUI is running on the suite host account, but full
functionality for remote suites requires either a shared filesystem, or 
(see~\ref{RemoteControl}) auth file installation {\em and} non-interactive ssh
access to the suite host.  Without the auth files you will not be able to connect
to the suite, and without ssh you will see ``permission denied'' errors on
attempting file access.

\subsection{Remote Control}
\label{RemoteControl}

Cylc client programs - command line and GUI - can interact with suite server
programs running on other accounts or hosts. How this works depends on whether
or not you have:
\begin{myitemize}
  \item a {\em shared filesystem} such that you see the same home directory on
    both hosts.
  \item {\em non-interactive ssh} from the client account to the server
    account.
\end{myitemize}

With a shared filesystem, a suite registered on the remote (server) host is
also - in effect - registered on the local (client) host.  In this case you
can invoke client commands without the \lstinline=--host= option; the client
will automatically read the host and port from the contact file in the
suite service directory. 

To control suite server programs running under other user accounts or on other
hosts without a shared filesystem, the suite SSL certificate and passphrase
must be installed under your \lstinline=$HOME/.cylc/= directory:
\lstset{language=transcript}
\begin{lstlisting}
$HOME/.cylc/auth/OWNER@HOST/SUITE/
      ssl.cert
      passphrase
      contact  # (optional - see below)
\end{lstlisting}
where \lstinline=OWNER@HOST= is the suite host account and \lstinline=SUITE=
is the suite name. Client commands should then be invoked with the
\lstinline=--user= and \lstinline=--host= options, e.g.:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc gui --user=OWNER --host=HOST SUITE
\end{lstlisting}

Note remote suite auth files do not need to be installed for read-only access -
see~\ref{PublicAccess} - via the GUI or monitor.

The suite contact file (see~\ref{The Suite Contact File}) is not needed if 
you have read-access to the remote suite run directory via the local
filesystem or non-interactive ssh to the suite host account - client commands
will automatically read it. If you do install the contact file in your auth
directory note that the port number will need to be updated if the suite gets
restarted on a different port. Otherwise use \lstinline=cylc scan= to determine
the suite port number and use the \lstinline=--port= client command option.

{\em WARNING: possession of a suite passphrase gives full control over the
target suite, including {\em edit run} functionality - which lets you run
arbitrary scripting on job hosts as the suite owner. Further,
non-interactive ssh gives full access to the target user account, so we
recommended that this is only used to interact with suites running on
accounts to which you already have full access. }

\subsection{Scan And Gscan}
\label{Scan And Gscan}

Both \lstinline=cylc scan= and the \lstinline=cylc gscan= GUI can display
suites owned by other users on other hosts, including task state totals if the
public access level permits that (see~\ref{PublicAccess}). Clicking on a remote
suite in \lstinline=gscan= will open a \lstinline=cylc gui= to connect to that
suite. This will give you full control, if you have the suite auth files
installed; or it will display full read only information if the public access
level allows that.

\subsection{Task States Explained}

As a suite runs, its task proxies may pass through the following states:

\begin{myitemize}
  \item {\bf waiting} - still waiting for prerequisites (e.g.\ dependence on
    other tasks, and clock triggers) to be satisfied.

    \item {\bf held} - will not be submitted to run even if all prerequisites
      are satisfied, until released/un-held.

    \item {\bf queued} - ready to run (prerequisites satisfied) but
    temporarily held back by an {\em internal cylc queue}
    (see~\ref{InternalQueues}).

    \item {\bf ready} - ready to run (prerequisites satisfied) and
    handed to cylc's job submission sub-system.

    \item {\bf submitted} - submitted to run, but not executing yet
    (could be waiting in an external batch scheduler queue).

    \item {\bf submit-failed} - job submission failed {\em or}
    submitted job killed (cancelled) before commencing execution.

    \item {\bf submit-retrying} - job submission failed, but a submission retry
    was configured. Will only enter the {\em submit-failed} state if all
    configured submission retries are exhausted.

    \item {\bf running} - currently executing (a {\em task started}
    message was received, or the task polled as running).

    \item {\bf succeeded} - finished executing successfully (a {\em task
    succeeded} message was received, or the task polled as succeeded).

    \item {\bf failed} - aborted execution due to some error condition (a
    {\em task failed} message was received, or the task polled as failed).

    \item {\bf retrying} - job execution failed, but an execution retry
    was configured. Will only enter the {\em failed} state if all configured
    execution retries are exhausted.

    \item {\bf runahead} - will not have prerequisites checked (and so
      automatically held, in effect) until the rest of the suite catches up
      sufficiently.  The amount of runahead allowed is configurable - see
      ~\ref{RunaheadLimit}.

    \item {\bf expired} - will not be submitted to run, due to falling too far
      behind the wall-clock relative to its cycle point -
      see~\ref{ClockExpireTasks}.

\end{myitemize}

\subsection{What The Suite Control GUI Shows}

The GUI Text-tree and Dot Views display the state of every task proxy present
in the task pool. Once a task has succeeded and Cylc has determined that it can
no longer be needed to satisfy the prerequisites of other tasks, its proxy will
be cleaned up (removed from the pool) and it will disappear from the GUI. To
rerun a task that has disappeared from the pool, you need to re-insert its task
proxy and then re-trigger it.

The Graph View is slightly different: it displays the complete dependency graph
over the range of cycle points currently present in the task pool. This often
includes some greyed-out {\em base} or {\em ghost nodes} that are empty - i.e.\
there are no corresponding task proxies currently present in the pool. Base
nodes just flesh out the graph structure. Groups of them may be cut out and
replaced by single {\em scissor nodes} in sections of the graph that are
currently inactive.


\subsection{Network Connection Timeouts}

A connection timeout can be set in site and user global config files
(see~\ref{SiteAndUserConfiguration}) so that messaging commands
cannot hang indefinitely if the suite is not responding (this can be
caused by suspending a suite with Ctrl-Z) thereby preventing the task
from completing. The same can be done on the command line for other
suite-connecting user commands, with the \lstinline=--comms-timeout= option.

\subsection{Runahead Limiting}
\label{RunaheadLimit}

Runahead limiting prevents the fastest tasks in a suite from getting too far
ahead of the slowest ones. Newly spawned tasks are released to the task pool
only when they fall below the runahead limit. A low runhead limit can prevent
cylc from interleaving cycles, but it will not stall a suite unless it fails to
extend out past a future trigger (see~\ref{InterCyclePointTriggers}).
A high runahead limit may allow fast tasks that are not constrained by
dependencies or clock-triggers to spawn far ahead of the pack, which could have
performance implications for the suite server program when running very large
suites.  Succeeded and failed tasks are ignored when computing the runahead
limit.

The preferred runahead limiting mechanism restricts the number of consecutive
active cycle points. The default value is three active cycle points;
see~\ref{max active cycle points}. Alternatively the interval between the
slowest and fastest tasks can be specified as hard limit;
see~\ref{runahead limit}.

\subsection{Limiting Activity With Internal Queues}
\label{InternalQueues}

Large suites can potentially overwhelm task hosts by submitting too many
tasks at once. You can prevent this with {\em internal queues}, which
limit the number of tasks that can be active (submitted or running)
at the some time.

A queue is defined by a {\em name}; a {\em limit}, which is the maximum
number of active tasks allowed for the queue; and a list of {\em members},
assigned by task or family name.

Queue configuration is done under the [scheduling] section of the suite.rc file
(like dependencies, internal queues constrain {\em when} a task runs).

By default every task is assigned to the {\em default} queue, which by default
has a zero limit (interpreted by cylc as no limit). To use a single queue for
the whole suite just set the default queue limit:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    [[ queues]]
        # limit the entire suite to 5 active tasks at once
        [[[default]]]
            limit = 5
\end{lstlisting}
To use additional queues just name each one, set their limits, and assign
members:
\begin{lstlisting}
[scheduling]
    [[ queues]]
        [[[q_foo]]]
            limit = 5
            members = foo, bar, baz
\end{lstlisting}
Any tasks not assigned to a particular queue will remain in the default
queue. The {\em queues} example suite illustrates how queues work by
running two task trees side by side (as seen in the graph GUI) each
limited to 2 and 3 tasks respectively:
\lstset{language=suiterc}
\lstinputlisting{../../../examples/queues/suite.rc}

\subsection{Automatic Task Retry On Failure}
\label{TaskRetries}

See also~\ref{RefRetries} in the {\em Suite.rc Reference}.

Tasks can be configured with a list of ``retry delay'' intervals, as
ISO 8601 durations. If the task job fails it will go into the {\em retrying}
state and resubmit after the next configured delay interval. An example is
shown in the suite listed below under~\ref{EventHandling}.

If a task with configured retries is {\em killed} (by \lstinline=cylc kill= or
via the GUI) it goes to the {\em held} state so that the operator can decide
whether to release it and continue the retry sequence or to abort the retry
sequence by manually resetting it to the {\em failed} state.

\subsection{Task Event Handling}
\label{EventHandling}

See also~\ref{SuiteEventHandling} and~\ref{TaskEventHandling} in the {\em
Suite.rc Reference}.

Cylc can call nominated event handlers - to do whatever you like - when certain
suite or task events occur. This facilitates centralized alerting and automated
handling of critical events. Event handlers can be used to send a message, call
a pager, or whatever; they can even intervene in the operation of their own
suite using cylc commands.

To send an email, use the built-in setting \lstinline=[[[events]]]mail events=
to specify a list of events for which notifications should be sent. E.g.\ to
send an email on (submission) failed and retry:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        script = "test ${CYLC_TASK_TRY_NUMBER} -eq 3"
        [[[events]]]
            mail events = submission failed, submission retry, failed, retry
        [[[job]]]
            execution retry delays = PT0S, PT30S
\end{lstlisting}

By default, the emails will be sent to the current user with:

\begin{myitemize}
    \item \lstinline=to:= set as \lstinline=$USER=
    \item \lstinline=from:= set as \lstinline=notifications@$(hostname)=
    \item SMTP server at \lstinline=localhost:25=
\end{myitemize}

These can be configured using the settings:
\begin{myitemize}
    \item \lstinline=[[[events]]]mail to= (list of email addresses),
    \item \lstinline=[[[events]]]mail from=
    \item \lstinline=[[[events]]]mail smtp=.
\end{myitemize}

By default, a cylc suite will send you no more than one task event email every
5 minutes - this is to prevent your inbox from being flooded by emails should a
large group of tasks all fail at similar time.
See ~\ref{task-event-mail-interval} for details.

Event handlers can be located in the suite \lstinline=bin/= directory;
otherwise it is up to you to ensure their location is in \lstinline=$PATH= (in
the shell in which the suite server program runs). They should require little
resource and return quickly - as each event handler is invoked by a child
process in a finite process pool that is also used to submit, poll and kill
jobs. The child process will wait for the event handler to complete before
moving on to the next item in the queue. If the process pool is saturated with
long running event handlers, the suite will appear to hang.

Task event handlers can be specified using the
\lstinline=[[[events]]]<event> handler= settings, where
\lstinline=<event>= is one of:
\begin{myitemize}
    \item `submitted' - the job submit command was successful
    \item `submission failed' - the job submit command failed
    \item `submission timeout' - task job submission timed out
    \item `submission retry' - task job submission failed, but will retry after
      a configured delay
    \item `started' - the task reported commencement of execution
    \item `succeeded' - the task reported successful completion
    \item `warning' - the task reported a WARNING severity message
    \item `critical' - the task reported a CRITICAL severity message
    \item `custom' - the task reported a CUSTOM severity message
    \item `failed' - the task failed
    \item `retry' - the task failed but will retry after a configured delay
    \item `execution timeout' - task execution timed out
\end{myitemize}

The value of each setting should be a list of command lines or command line
templates (see below).

Alternatively you can use \lstinline=[[[events]]]handlers= and
\lstinline=[[[events]]]handler events=, where the former is a list of command
lines or command line templates (see below) and the latter is a list of events
for which these commands should be invoked.

Event handler arguments can be constructed from various templates
representing suite name; task ID, name, cycle point, message, and submit
number name; and any suite or task [meta] item. See~\ref{SuiteEventHandling}
and~\ref{TaskEventHandling} for options.

If no template arguments are supplied the following default command line
will be used:
\begin{lstlisting}
<task-event-handler> %(event)s %(suite)s %(id)s %(message)s
\end{lstlisting}

{\em Note: substitution patterns should not be quoted in the template strings.
This is done automatically where required.}

For an explanation of the substitution syntax, see
\href{https://docs.python.org/2/library/stdtypes.html#string-formatting}{String Formatting Operations}
in the Python documentation.

The retry event occurs if a task fails and has any remaining retries
configured (see~\ref{TaskRetries}).
The event handler will be called as soon as the task fails, not after
the retry delay period when it is resubmitted.

{\em Note that event handlers are called by the suite server program, not by task jobs.}
If you wish to pass additional information to them use [cylc] \textrightarrow
[[environment]], not task runtime environment.

The following 2 \lstinline=suite.rc= snippets are examples on how to specify
event handlers using the alternate methods:

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        script = "test ${CYLC_TASK_TRY_NUMBER} -eq 2"
        [[[events]]]
            retry handler = "echo '!!!!!EVENT!!!!!' "
            failed handler = "echo '!!!!!EVENT!!!!!' "
        [[[job]]]
            execution retry delays = PT0S, PT30S
\end{lstlisting}

\lstset{language=suiterc}
\begin{lstlisting}
[runtime]
    [[foo]]
        script = "test ${CYLC_TASK_TRY_NUMBER} -eq 2"
        [[[events]]]
            handlers = "echo '!!!!!EVENT!!!!!' "
            handler events = retry, failed
        [[[job]]]
            execution retry delays = PT0S, PT30S
\end{lstlisting}
The handler command here - specified with no arguments - is called with the
default arguments, like this:
\begin{lstlisting}
echo '!!!!!EVENT!!!!!' %(event)s %(suite)s %(id)s %(message)s
\end{lstlisting}


\subsection{Handling Job Preemption}
\label{PreemptionHPC}

Some HPC facilities allow job preemption: the resource manager can kill
or suspend running low priority jobs in order to make way for high
priority jobs. The preempted jobs may then be automatically restarted
by the resource manager, from the same point (if suspended) or requeued
to run again from the start (if killed).

Suspended jobs will poll as still running (their job status file says they
started running, and they still appear in the resource manager queue).
Loadleveler jobs that are preempted by kill-and-requeue ("job vacation") are
automatically returned to the submitted state by Cylc.  This is possible
because Loadleveler sends the SIGUSR1 signal before SIGKILL for preemption.
Other batch schedulers just send SIGTERM before SIGKILL as normal, so Cylc
cannot distinguish a preemption job kill from a normal job kill. After this the
job will poll as failed (correctly, because it was killed, and the job status
file records that). To handle this kind of preemption automatically you could
use a task failed or retry event handler that queries the batch scheduler queue
(after an appropriate delay if necessary) and then, if the job has been
requeued, uses \lstinline=cylc reset= to reset the task to the submitted state.

\subsection{Manual Task Triggering and Edit-Run}

Any task proxy currently present in the suite can be manually triggered at any
time using the \lstinline=cylc trigger= command, or from the right-click task
menu in gcylc. If the task belongs to a limited internal queue
(see~\ref{InternalQueues}), this will queue it; if not, or if it is already
queued, it will submit immediately.

With \lstinline=cylc trigger --edit= (also in the gcylc right-click task menu)
you can edit the generated task job script to make one-off changes before the
task submits.

\subsection{Cylc Broadcast}
\label{cylc-broadcast}

The \lstinline=cylc broadcast= command overrides \lstinline=[runtime]=
settings in a running suite. This can
be used to communicate information to downstream tasks by broadcasting
environment variables (communication of information from one task to
another normally takes place via the filesystem, i.e.\ the input/output
file relationships embodied in inter-task dependencies). Variables (and
any other runtime settings) may be broadcast to all subsequent tasks,
or targeted specifically at a specific task, all subsequent tasks with a
given name, or all tasks with a given cycle point; see broadcast command help
for details.

Broadcast settings targeted at a specific task ID or cycle point expire and
are forgotten as the suite moves on. Un-targeted variables and those
targeted at a task name persist throughout the suite run, even across
restarts, unless manually cleared using the broadcast command - and so
should be used sparingly.

\subsection{The Meaning And Use Of Initial Cycle Point}

When a suite is started with the \lstinline=cylc run= command (cold or
warm start) the cycle point at which it starts can be given on the command
line or hardwired into the suite.rc file:
\begin{lstlisting}
cylc run foo 20120808T06Z
\end{lstlisting}
or:
\begin{lstlisting}
[scheduling]
    initial cycle point = 20100808T06Z
\end{lstlisting}
An initial cycle given on the command line will override one in the
suite.rc file.

\subsubsection[CYLC\_SUITE\_INITIAL\_CYCLE\_POINT]{The Environment Variable CYLC\_SUITE\_INITIAL\_CYCLE\_POINT}

In the case of a {\em cold start only} the initial cycle point is passed
through to task execution environments as
\lstinline=$CYLC_SUITE_INITIAL_CYCLE_POINT=. The value is then stored in
suite database files and persists across restarts, but it does get wiped out
(set to \lstinline=None=) after a warm start, because a warm start is really an
implicit restart in which all state information is lost (except that the
previous cycle is assumed to have completed).

The \lstinline=$CYLC_SUITE_INITIAL_CYCLE_POINT= variable allows tasks to
determine if they are running in the initial cold-start cycle point, when
different behaviour may be required, or in a normal mid-run cycle point.
Note however that an initial \lstinline=R1= graph section is now the preferred
way to get different behaviour at suite start-up.

\subsection{Simulating Suite Behaviour}
\label{SimulationMode}

Several suite run modes allow you to simulate suite behaviour quickly without
running the suite's real jobs - which may be long-running and resource-hungry:

\begin{myitemize}
  \item {\em dummy mode} - runs dummy tasks as background jobs on configured
    job hosts.
    \begin{myitemize}
      \item simulates scheduling, job host connectivity, and
        generates all job files on suite and job hosts.
    \end{myitemize}
  \item {\em dummy-local mode} - runs real dummy tasks as background jobs on
    the suite host, which allows dummy-running suites from other sites.
    \begin{myitemize}
      \item simulates scheduling and generates all job files on the
        suite host.
    \end{myitemize}
  \item {\em simulation mode} - does not run any real tasks.
    \begin{myitemize}
      \item simulates scheduling without generating any job files.
    \end{myitemize}
\end{myitemize}

Set the run mode (default {\em live}) in the GUI suite start dialog box, or on
the command line:
\lstset{language=transcript}
\begin{lstlisting}
$ cylc run --mode=dummy SUITE
$ cylc restart --mode=dummy SUITE
\end{lstlisting}

You can get specified tasks to fail in these modes, for more flexible suite
testing. See Section~\ref{suiterc-sim-config} for simulation configuration.

\subsubsection{Proportional Simulated Run Length}

If task \lstinline=[job]execution time limit= is set, Cylc divides it by
\lstinline=[simulation]speedup factor= (default \lstinline=10.0=) to compute
simulated task run lengths (default 10 seconds).

\subsubsection{Limitations Of Suite Simulation}

Dummy mode ignores batch scheduler settings because Cylc does not know which
job resource directives (requested memory, number of compute nodes, etc.) would
need to be changed for the dummy jobs.  If you need to dummy-run jobs on a
batch scheduler manually comment out \lstinline=script= items and modify
directives in your live suite, or else use a custom live mode test suite.

Note that the dummy modes ignore all configured task \lstinline=script= items
including \lstinline=init-script=. If your \lstinline=init-script= is required
to run even dummy tasks on a job host, note that host environment setup should
be done elsewhere - see~\ref{Configure Site Environment on Job Hosts}.

\subsubsection{Restarting Suites With A Different Run Mode?}

The run mode is recorded in the suite run database files. Cylc will not let
you {\em restart} a non-live mode suite in live mode, or vice versa. To
test a live suite in simulation mode just take a quick copy of it and run the
the copy in simulation mode.

\subsection{Automated Reference Test Suites}
\label{AutoRefTests}

Reference tests are finite-duration suite runs that abort with non-zero
exit status if any of the following conditions occur (by default):

\begin{myitemize}
    \item cylc fails
    \item any task fails
    \item the suite times out (e.g.\ a task dies without reporting failure)
    \item a nominated shutdown event handler exits with error status
\end{myitemize}

The default shutdown event handler for reference tests is
\lstinline=cylc hook check-triggering= which compares task triggering
information (what triggers off what at run time) in the test run suite
log to that from an earlier reference run, disregarding the timing and
order of events - which can vary according to the external queueing
conditions, runahead limit, and so on.

To prepare a reference log for a suite, run it with the
\lstinline=--reference-log= option, and manually verify the
correctness of the reference run.

To reference test a suite, just run it (in dummy mode for the most
comprehensive test without running real tasks) with the
\lstinline=--reference-test= option.

A battery of automated reference tests is used to test cylc before
posting a new release version. Reference tests can also be used to check that
a cylc upgrade will not break your own complex
suites - the triggering check will catch any bug that causes a task to
run when it shouldn't, for instance; even in a dummy mode reference
test the full task job script (sans \lstinline=script= items) executes on the
proper task host by the proper batch system.

Reference tests can be configured with the following settings:
\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    [[reference test]]
        suite shutdown event handler = cylc check-triggering
        required run mode = dummy
        allow task failures = False
        live mode suite timeout = PT5M
        dummy mode suite timeout = PT2M
        simulation mode suite timeout = PT2M
\end{lstlisting}

\subsubsection{Roll-your-own Reference Tests}

If the default reference test is not sufficient for your needs, firstly
note that you can override the default shutdown event handler, and
secondly that the \lstinline=--reference-test= option is merely a short
cut to the following suite.rc settings which can also be set manually if
you wish:

\lstset{language=suiterc}
\begin{lstlisting}
[cylc]
    abort if any task fails = True
    [[events]]
        shutdown handler = cylc check-triggering
        timeout = PT5M
        abort if shutdown handler fails = True
        abort on timeout = True
\end{lstlisting}

\subsection{Triggering Off Of Tasks In Other Suites}
\label{SuiteStatePolling}

The \lstinline=cylc suite-state= command interrogates suite run databases. It
has a polling mode that waits for a given task in the target suite to achieve a
given state. This can be used to make task scripting wait for a remote task
to succeed (for example). The suite graph notation also provides a way to
define automatic suite-state polling tasks, which use the same polling command
under the hood. Note that cylc suite-state can only trigger off task
{\em states} in remote suites and does not support triggering off task
messages.

Here's how to trigger a task \lstinline=bar= off a task \lstinline=foo= in
a remote suite called \lstinline=other.suite=:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00, T12]]]
            graph = "my-foo<other.suite::foo> => bar"
\end{lstlisting}
Local task \lstinline=my-foo= will poll for the success of \lstinline=foo=
in suite \lstinline=other.suite=, at the same cycle point, succeeding only when
or if it succeeds. Other task states can also be polled:
\begin{lstlisting}
   graph = "my-foo<other.suite::foo:fail> => bar"
\end{lstlisting}

The default polling parameters (e.g.\ maximum number of polls and the interval
between them) are printed by \lstinline=cylc suite-state --help= and can be
configured if necessary under the local polling task runtime section:
\begin{lstlisting}
[scheduling]
    [[ dependencies]]
        [[[T00,T12]]]
            graph = "my-foo<other.suite::foo> => bar"
[runtime]
    [[my-foo]]
        [[[suite state polling]]]
            max-polls = 100
            interval = PT10S
\end{lstlisting}

For suites owned by others, or those with run databases in non-standard
locations, use the \lstinline=--run-dir= option, or in-suite:
\begin{lstlisting}
[runtime]
    [[my-foo]]
        [[[suite state polling]]]
            run-dir = /path/to/top/level/cylc/run-directory
\end{lstlisting}

If the remote task has a different cycling sequence, just arrange for the
local polling task to be on the same sequence as the remote task that it
represents. For instance, if local task \lstinline=cat= cycles 6-hourly at
\lstinline=0,6,12,18= but needs to trigger off a remote task \lstinline=dog=
at \lstinline=3,9,15,21=:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T03,T09,T15,T21]]]
            graph = "my-dog<other.suite::dog>"
        [[[T00,T06,T12,T18]]]
            graph = "my-dog[-PT3H] => cat"
\end{lstlisting}

For suite-state polling the cycle point of the target task is treated as a
literal string so the polling command has to be told if the remote suite has a
different cycle point format. Use the \lstinline=--template= option for this,
or in-suite:
\begin{lstlisting}
[runtime]
    [[my-foo]]
        [[[suite state polling]]]
            template = %Y-%m-%dT%H
\end{lstlisting}

Note that the remote suite does not have to be running when polling commences
because the command interrogates the suite run database, not the suite server
process.

\subsection{Suite Server Logs}
\label{Suite Server Logs}

Each suite maintains its own log of time-stamped events under the {\em suite
server log directory}:

\begin{lstlisting}
$HOME/cylc-run/SUITE-NAME/log/suite/
\end{lstlisting}

By way of example, we will show the complete server log generated (at
cylc-7.2.0) by a small suite that runs two 30-second dummy tasks
\lstinline=foo= and \lstinline=bar= for a single cycle point
\lstinline=2017-01-01T00Z= before shutting down:

\lstset{language=suiterc,breaklines=true}
\begin{lstlisting}
[cylc]
    cycle point format = %Y-%m-%dT%HZ
[scheduling]
    initial cycle point = 2017-01-01T00Z
    final cycle point = 2017-01-01T00Z
    [[dependencies]]
        graph = "foo => bar"
[runtime]
    [[foo]]
        script = sleep 30; /bin/false
    [[bar]]
        script = sleep 30; /bin/true
\end{lstlisting}

By the task scripting defined above, this suite will stall when \lstinline=foo=
fails. Then, the suite owner {\em vagrant@cylon} manually resets the failed
task's state to {\em succeeded}, allowing \lstinline=bar= to trigger and the
suite to finish and shut down.  Here's the complete suite log for this run:

\lstset{language=transcript}
\begin{lstlisting}
$ cylc cat-log SUITE-NAME
2017-03-30T09:46:10Z INFO - Suite starting: server=localhost:43086 pid=3483
2017-03-30T09:46:10Z INFO - Run mode: live
2017-03-30T09:46:10Z INFO - Initial point: 2017-01-01T00Z
2017-03-30T09:46:10Z INFO - Final point: 2017-01-01T00Z
2017-03-30T09:46:10Z INFO - Cold Start 2017-01-01T00Z
2017-03-30T09:46:11Z INFO - [foo.2017-01-01T00Z] -submit_method_id=3507
2017-03-30T09:46:11Z INFO - [foo.2017-01-01T00Z] -submission succeeded
2017-03-30T09:46:11Z INFO - [foo.2017-01-01T00Z] -(current:submitted)> started at 2017-03-30T09:46:10Z
2017-03-30T09:46:41Z CRITICAL - [foo.2017-01-01T00Z] -(current:running)> Task job script received signal EXIT at 2017-03-30T09:46:40Z
2017-03-30T09:46:41Z CRITICAL - [foo.2017-01-01T00Z] -(current:running)> failed at 2017-03-30T09:46:40Z
2017-03-30T09:46:42Z WARNING - suite stalled
2017-03-30T09:46:42Z WARNING - Unmet prerequisites for bar.2017-01-01T00Z:
2017-03-30T09:46:42Z WARNING -  * foo.2017-01-01T00Z succeeded
2017-03-30T09:47:58Z INFO - [client-command] reset_task_states vagrant@cylon:cylc-reset 1e0d8e9f-2833-4dc9-a0c8-9cf263c4c8c3
2017-03-30T09:47:58Z INFO - [foo.2017-01-01T00Z] -resetting state to succeeded
2017-03-30T09:47:58Z INFO - Command succeeded: reset_task_states([u'foo.2017'], state=succeeded)
2017-03-30T09:47:59Z INFO - [bar.2017-01-01T00Z] -submit_method_id=3565
2017-03-30T09:47:59Z INFO - [bar.2017-01-01T00Z] -submission succeeded
2017-03-30T09:47:59Z INFO - [bar.2017-01-01T00Z] -(current:submitted)> started at 2017-03-30T09:47:58Z
2017-03-30T09:48:29Z INFO - [bar.2017-01-01T00Z] -(current:running)> succeeded at 2017-03-30T09:48:28Z
2017-03-30T09:48:30Z INFO - Waiting for the command process pool to empty for shutdown
2017-03-30T09:48:30Z INFO - Suite shutting down - AUTOMATIC
\end{lstlisting}

The information logged here includes:

\begin{myitemize}
  \item event timestamps, at the start of each line
  \item suite server host, port and process ID
  \item suite initial and final cycle points
  \item suite start type (cold start in this case)
  \item task events (task started, succeeded, failed, etc.)
  \item suite stalled warning (in this suite nothing else can run when
    \lstinline=foo= fails)
  \item the client command issued by {\em vagrant@cylon} to reset
    \lstinline=foo= to {\em succeeded}
  \item job IDs  - in this case process IDs for background jobs (or PBS job IDs
    etc.)
  \item state changes due to incoming task progress message  ("started at ..."
    etc.) suite shutdown time and reasons (AUTOMATIC means "all tasks finished
    and nothing else to do")
\end{myitemize}

Note that suite log files are primarily intended for human eyes. If you need
to have an external system to monitor suite events automatically, interrogate
the sqlite {\em suite run database} (see~\ref{Suite Run
Databases}) rather than parse the log files.

\subsection{Suite Run Databases}
\label{Suite Run Databases}

Suite server programs maintain two \lstinline=sqlite= databases to record
restart checkpoints and various other aspects of run history:

\lstset{language=transcript}
\begin{lstlisting}
$HOME/cylc-run/SUITE-NAME/log/db  # public suite DB
$HOME/cylc-run/SUITE-NAME/.service/db  # private suite DB
\end{lstlisting}

The private DB is for use only by the suite server program. The identical
public DB is provided for use by external commands such as
\lstinline=cylc suite-state=, \lstinline=cylc ls-checkpoints=, and
\lstinline=cylc report-timings=. If the public DB gets locked for too long by
an external reader, the suite server program will eventually delete it and
replace it with a new copy of the private DB, to ensure that both correctly
reflect the suite state.

You can interrogate the public DB with the \lstinline=sqlite3= command line tool,
the \lstinline=sqlite3= module in the Python standard library, or any other
sqlite interface.

\begin{lstlisting}
$ sqlite3 ~/cylc-run/foo/log/db << _END_
> .headers on
> select * from task_events where name is "foo";
> _END_
name|cycle|time|submit_num|event|message
foo|1|2017-03-12T11:06:09Z|1|submitted|
foo|1|2017-03-12T11:06:09Z|1|output completed|started
foo|1|2017-03-12T11:06:09Z|1|started|
foo|1|2017-03-12T11:06:19Z|1|output completed|succeeded
foo|1|2017-03-12T11:06:19Z|1|succeeded|
\end{lstlisting}

\subsection{Disaster Recovery}
\label{Disaster Recovery}

If a suite run directory gets deleted or corrupted, the options for recovery
are:
\begin{myitemize}
  \item restore the run directory from back-up, and restart the suite
  \item re-install from source, and warm start from the beginning of the
    current cycle point
\end{myitemize}

A warm start (see~\ref{Warm Start}) does not need a suite state checkpoint, but
it wipes out prior run history, and it could re-run a significant number of
tasks that had already completed. 

To restart the suite, the critical Cylc files that must be restored are:

\lstset{language=transcript}
\begin{lstlisting}
# On the suite host:
~/cylc-run/SUITE-NAME/
    suite.rc   # live suite definition (located here in Rose suites)
    log/db  # public suite DB (can just be a copy of the private DB)
    log/rose-suite-run.conf  # (needed to restart a Rose suite)
    .service/db  # private suite DB
    .service/source -> PATH-TO-SUITE-DIR  # symlink to live suite directory
 
# On job hosts (if no shared filesystem):
~/cylc-run/SUITE-NAME/
    log/job/CYCLE-POINT/TASK-NAME/SUBMIT-NUM/job.status
\end{lstlisting}

{\em Note this discussion does not address restoration of files generated and
consumed by task jobs at run time}. How suite data is stored and recovered in
your environment is a matter of suite and system design.

In short, you can simply restore the suite service directory, the log
directory, and the suite.rc file that is the target of the symlink in the
service directory. The service and log directories will come with extra files
that aren't strictly needed for a restart, but that doesn't matter - although
depending on your log housekeeping the \lstinline=log/job= directory could be
huge, so you might want to be selective about that.  (Also in a Rose suite, the
\lstinline=suite.rc= file does not need to be restored if you restart with
\lstinline=rose suite-run= - which re-installs suite source files to the run
directory).

The public DB is not strictly required for a restart - the suite server program
will recreate it if need be - but it is required by
\lstinline=cylc ls-checkpoints= if you need to identify the right restart
checkpoint.

The job status files are only needed if the restart suite state checkpoint
contains active tasks that need to be polled to determine what happened to them
while the suite was down.  Without them, polling will fail and those tasks will
need to be manually set to the correct state.

{\em WARNING: it is not safe to copy or rsync a potentially-active sqlite DB -
the copy might end up corrupted. It is best to stop the suite before copying
a DB, or else write a back-up utility using the official sqlite backup API:
\url{http://www.sqlite.org/backup.html}.}


\section{Suite Storage, Discovery, Revision Control, and Deployment}
\label{SuiteStorageEtc}

Small groups of cylc users can of course share suites by manual copying,
and generic revision control tools can be used on cylc suites as for any
collection of files. Beyond this cylc does not have a built-in solution
for suite storage and discovery, revision control, and deployment, on a
network. That is not cylc's core purpose, and large sites may have
preferred revision control systems and suite meta-data requirements that
are difficult to anticipate. We can, however, recommend the use of {\em
Rose} to do all of this very easily and elegantly with cylc suites.

\subsection{Rose}
\label{Rose}

{\bf Rose} is {\em a framework for managing and running suites of
scientific applications}, developed at the UK Met Office for use with
cylc. It is available under the open source GPL license.

\begin{myitemize}
    \item Rose documentation: \url{http://metomi.github.io/rose/doc/rose.html}
    \item Rose source repository: \url{https://github.com/metomi/rose}
\end{myitemize}

\pagebreak

\appendix

\input{suiterc.tex}
\pagebreak

\input{siterc.tex}

\pagebreak

\input{gcylcrc.tex}

\pagebreak

\input{gscanrc.tex}

\pagebreak

\input{job-host.tex}

\pagebreak


\section{Command Reference}
\label{CommandReference}

%This section is auto-generated from the self-documenting command set.

\lstset{language=usage}
\input{commands.tex}
\lstset{language=transcript}

\section{The gcylc Graph View}
\label{TheGraphBasedcontrolGUI}

The graph view in the gcylc GUI shows the structure of the suite as it
evolves. It can work well even for large suites, but be aware that the
graphviz layout engine has to do a new global layout every time a task
proxy appears in or disappears from the task pool. The following may help
mitigate any jumping layout problems:

\begin{myitemize}
    \item The disconnect button can be used to temporarily prevent the
        graph from changing as the suite evolves.
    \item The greyed-out base nodes, which are only present to fill out
        the graph structure, can be toggled off (but this will split the
        graph into disconnected sub-trees).
    \item Right-click on a task and choose the ``Focus'' option to restrict
        the graph display to that task's cycle point. Anything interesting
        happening in other cycle points will show up as disconnected
        rectangular nodes to the right of the graph (and you can click on
        those to instantly refocus to their cycle points).
    \item Task filtering is the ultimate quick route to focusing on just
        the tasks you're interested in, but this will destroy the graph
        structure.
\end{myitemize}

\section{Cylc README File}

\lstinputlisting{../../../README.md}

\section{Cylc INSTALL File}
\label{INSTALL}

\lstinputlisting{../../../INSTALL.md}

\section{Cylc Development History - Major Changes}

\begin{myitemize}

    \item {\bf pre-cylc-3} - early versions focused on the new
    scheduling algorithm. A suite was a collection of ``task definition
    files'' that encoded the prerequisites and outputs of each task,
    exposing cylc's self-organising nature. Tasks could be transferred
    from one suite to another by simply copying their taskdef files over
    and checking prerequisite and output consistency. Global suite
    structure was not easy to discern until run time (although cylc-2
    could generate resolved run time dependency graphs).

    \item {\bf cylc-3} - a new suite design interface: dependency graph
    and task runtime properties defined in a single structured,
    validated, configuration file - the suite.rc file; graphical user
    interface; suite graphing.

    \item {\bf cylc-4} - refined and organized the suite.rc file
    structure; task runtime properties defined by an efficient
    inheritance hierarchy; support for the Jinja2 template processor in
    suite definitions.

    \item {\bf cylc-5} - multi-threading for continuous network request
    handling and job submission; more task states to distinguish job
    submission from execution; dependence between suites via new suite
    run databases; polling and killing of real task jobs; polling as
    task communications option.

    \item {\bf cylc-6} - specification of all date-times and cycling
    workflows via the ISO8601 date-times, durations, and recurrence
    expressions; integer cycling; a multi-process pool to execute job
    submissions, event handlers, and poll and kill commands.

    \item {\bf cylc-7} - Replaced the Pyro communications layer with
     RESTful HTTPS. Removed deprecated pre cylc-6 syntax and features.

\end{myitemize}

\section{Communication Method}
\label{Communication}

Cylc suite server programs and clients (commands, cylc gui, task messaging)
communicate via particular ports using the HTTPS protocol, secured
by HTTP Digest Authentication using the suite's 20-random-character
private passphrase and private SSL certificate.

This is enabled via the included-in-cylc cherrypy library (for the
server) and either the Python requests library (if available) or
the built-in Python libraries for the clients.

All suites are entirely isolated from one another.

\section{Cylc 6 Migration Reference}
\label{cylc-6-migration}

Cylc 6 introduced new date-time-related syntax for the suite.rc file. In
some places, this is quite radically different from the earlier syntax.

\subsection{Timeouts and Delays}
\label{cylc-6-migration-timeout-delays}

Timeouts and delays such as \lstinline=[cylc][[events]]timeout= or
\lstinline=[runtime][[my_task]][[[job]]]execution retry delays= were written in
a purely numeric form before cylc 6, in seconds, minutes (most common), or
hours, depending on the setting.

They are now written in an ISO 8601 duration form, which has the benefit
that the units are user-selectable (use 1 day instead of 1440 minutes)
and explicit.

Nearly all timeouts and delays in cylc were in minutes, except for:\\*
\lstinline=[runtime][[my_task]][[[suite state polling]]]interval= \\*
\lstinline=[runtime][[my_task]][[[simulation mode]]]run time range= \\*
which were in seconds, and\\*
\lstinline=[scheduling]runahead limit=\\*
which was in hours (this is a special case discussed below
in~\ref{cylc-6-migration-runahead-limit}).

See Table \ref{cylc-6-migration-timeout-delays-table}.

\begin{table}[ht]
\caption{Timeout/Delay Syntax Change Examples}
\centering
\begin{tabular}{ l c c }
Setting & Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=[cylc][[events]]timeout= & 180 & PT3H \\
\lstinline=[runtime][[my_task]][[[job]]]execution retry delays= & 2*30, 360, & 2*PT30M, PT6H, \\
 & 1440 & P1D \\
\lstinline=[runtime][[my_task]][[[suite state polling]]]interval= & 2 & PT2S \\
\end{tabular}
\label{cylc-6-migration-timeout-delays-table}
\end{table}

\subsection{Runahead Limit}
\label{cylc-6-migration-runahead-limit}

See~\ref{runahead limit}.

The \lstinline=[scheduling]runahead limit= setting was written as a number of
hours in pre-cylc-6 suites. This is now in ISO 8601 format for date-time
cycling suites, so \lstinline@[scheduling]runahead limit=36@ would be written
\lstinline@[scheduling]runahead limit=PT36H@.

There is a new preferred alternative to \lstinline=runahead limit=,
\lstinline=[scheduling]max active cycle points=. This allows the user to
configure how many cycle points can run at once (default \lstinline=3=). See
\ref{max active cycle points}.

\subsection{Cycle Time/Cycle Point}
\label{cylc-6-migration-cycle-point}

See~\ref{initial cycle point}.

The following suite.rc settings have changed name (Table
\ref{cylc-6-migration-cycle-point-time-table}):

\begin{table}[ht]
\caption{Cycle Point Renaming}
\centering
\begin{tabular}{ l l }
Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=[scheduling]initial cycle time= & \lstinline=[scheduling]initial cycle point= \\
\lstinline=[scheduling]final cycle time= & \lstinline=[scheduling]final cycle point= \\
\lstinline=[visualization]initial cycle time= & \lstinline=[visualization]initial cycle point= \\
\lstinline=[visualization]final cycle time= & \lstinline=[visualization]final cycle point= \\
\end{tabular}
\label{cylc-6-migration-cycle-point-time-table}
\end{table}

This change is to reflect the fact that cycling in cylc 6+ can now be over
e.g.\ integers instead of being purely based on date-time.

Date-times written in \lstinline=initial cycle time= and
\lstinline=final cycle time= were in a cylc-specific 10-digit (or less)
\lstinline=CCYYMMDDhh= format, such as \lstinline=2014021400= for 00:00 on
the 14th of February 2014.

Date-times are now required to be ISO 8601 compatible. This can be achieved
easily enough by inserting a \lstinline=T= between the day and the hour
digits.

\begin{table}[ht]
\caption{Cycle Point Syntax Example}
\centering
\begin{tabular}{ l c c }
Setting & Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=[scheduling]initial cycle time= & 2014021400 & 20140214T00 \\
\end{tabular}
\label{cylc-6-migration-cycle-point-syntax-table}
\end{table}

\subsection{Cycling}
\label{cylc-6-migration-cycling}

Special {\em start-up} and {\em cold-start} tasks have been removed from cylc
6. Instead, use the initial/run-once notation as detailed
in~\ref{initial-non-repeating-r1-tasks} and~\ref{AdvancedStartingUp}.

{\em Repeating asynchronous tasks} have also been removed because non date-time
workflows can now be handled more easily with integer cycling. See for instance
the satellite data processing example documented in~\ref{IntegerCycling}.

For repeating tasks with hour-based cycling the syntax has only minor changes:

Pre-cylc-6:
\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    ...
    [[dependencies]]
        [[[0,12]]]
            graph = foo[T-12] => foo & bar => baz
\end{lstlisting}
\lstset{language=transcript}

\lstset{language=suiterc}
\begin{lstlisting}
[scheduling]
    ...
    [[dependencies]]
        [[[T00,T12]]]
            graph = foo[-PT12H] => foo & bar => baz
\end{lstlisting}
\lstset{language=transcript}

Hour-based cycling section names are easy enough to convert, as seen in Table
\ref{cylc-6-migration-cycling-hours-table}.

\begin{table}[ht]
\caption{Hourly Cycling Sections}
\centering
\begin{tabular}{ l l }
Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=[scheduling][[dependencies]][[[0]]]= & \lstinline=[scheduling][[dependencies]][[[T00]]]= \\
\lstinline=[scheduling][[dependencies]][[[6]]]= & \lstinline=[scheduling][[dependencies]][[[T06]]]= \\
\lstinline=[scheduling][[dependencies]][[[12]]]= & \lstinline=[scheduling][[dependencies]][[[T12]]]= \\
\lstinline=[scheduling][[dependencies]][[[18]]]= & \lstinline=[scheduling][[dependencies]][[[T18]]]= \\
\end{tabular}
\label{cylc-6-migration-cycling-hours-table}
\end{table}

The graph text in hour-based cycling is also easy to convert, as seen in
Table \ref{cylc-6-migration-cycling-hours-offset-table}.

\begin{table}[ht]
\caption{Hourly Cycling Offsets}
\centering
\begin{tabular}{ l l }
Pre-Cylc-6 & Cylc-6+ \\
\hline
\lstinline=my_task[T-6]= & \lstinline=my_task[-PT6H]= \\
\lstinline=my_task[T-12]= & \lstinline=my_task[-PT12H]= \\
\lstinline=my_task[T-24]= & \lstinline=my_task[-PT24H]= or even \lstinline=my_task[-P1D]= \\
\end{tabular}
\label{cylc-6-migration-cycling-hours-offset-table}
\end{table}

\subsection{No Implicit Creation of Tasks by Offset Triggers}
\label{cylc-6-migration-implicit-cycling}

Prior to cylc-6 intercycle offset triggers implicitly created task instances at
the offset cycle points. For example, this pre cylc-6 suite automatically
creates instances of task \lstinline=foo= at the offset hours
\lstinline=3,9,15,21= each day, for task \lstinline=bar= to trigger off at
\lstinline=0,6,12,18=:
\lstset{language=suiterc}
\begin{lstlisting}
# Pre cylc-6 implicit cycling.
[scheduling]
   initial cycle time = 2014080800
   [[dependencies]]
      [[[00,06,12,18]]]
         # This creates foo instances at 03,09,15,21:
         graph = foo[T-3] => bar
\end{lstlisting}

Here's the direct translation to cylc-6+ format:
\lstset{language=suiterc}
\begin{lstlisting}
# In cylc-6+ this suite will stall.
[scheduling]
   initial cycle point = 20140808T00
   [[dependencies]]
      [[[T00,T06,T12,T18]]]
         # This does NOT create foo instances at 03,09,15,21:
         graph = foo[-PT3H] => bar
\end{lstlisting}

This suite fails validation with
\lstinline=ERROR: No cycling sequences defined for foo=,
and at runtime it would stall with \lstinline=bar= instances waiting on
non-existent offset \lstinline=foo= instances (note that these
appear as ghost nodes in graph visualisations).

To fix this, explicitly define the cycling of with an offset cycling sequence:
\lstinline=foo=:
\lstset{language=suiterc}
\begin{lstlisting}
# Cylc-6+ requires explicit task instance creation.
[scheduling]
   initial cycle point = 20140808T00
   [[dependencies]]
      [[[T03,T09,T15,T21]]]
         graph = foo
      [[[T00,T06,T12,T18]]]
         graph = foo[-PT3H] => bar
\end{lstlisting}

Implicit task creation by offset triggers is no longer allowed because it is
error prone: a mistaken task cycle point offset should cause a failure
rather than automatically creating task instances on the wrong cycling
sequence.

\section{Known Issues}
\label{KnownIssues}

\subsection{Current Known Issues}
\label{CurrentKnownIssues}

The best place to find current known issues is on Github:
\url{https://github.com/cylc/cylc/issues}.

\subsection{Notable Known Issues}
\label{NotableKnownIssues}

\subsubsection{Use of pipes in job scripts}
\label{PipeInJobScripts}

In bash, the return status of a pipeline is normally the exit status of the
last command. This is unsafe, because if any command in the pipeline fails, the
script will continue nevertheless.

For safety, a cylc task job script running in bash will have the
\lstinline=set -o pipefail= option turned on automatically. If a pipeline
exists in a task's \lstinline=script=, etc section, the failure of any part of
a pipeline will cause the command to return a non-zero code at the end, which
will be reported as a task job failure. Due to the unique nature of a pipeline,
the job file will trap the failure of the individual commands, as well as the
whole pipeline, and will attempt to report a failure back to the suite twice.
The second message is ignored by the suite, and so the behaviour can be safely
ignored. (You should probably still investigate the failure, however!)

\section{GNU GENERAL PUBLIC LICENSE v3.0}
\input{gpl-3.0}
