.. _Tutorial:

Tutorial
========

This section provides a hands-on tutorial introduction to basic cylc
functionality.

User Config File
----------------

Some settings affecting cylc's behaviour can be defined in site and user
*global config files*. For example, to choose the text editor invoked by
cylc on suite configurations:

.. code-block:: cylc

   # $HOME/.cylc/$(cylc --version)/global.rc
   [editors]
       terminal = vim
       gui = gvim -f

- For more on site and user global config files
  see :ref:`SiteAndUserConfiguration` and :ref:`SiteRCReference`.


.. _Configure Environment on Job Hosts:

Configure Environment on Job Hosts
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

See :ref:`Configure Site Environment on Job Hosts` for information.


.. _CUI:

User Interfaces
---------------

You should have access to the cylc command line (CLI) and graphical (GUI) user
interfaces once cylc has been installed as described in
Section :ref:`InstallCylc`.

Command Line Interface (CLI)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The command line interface is unified under a single top level
``cylc`` command that provides access to many sub-commands
and their help documentation.

.. code-block:: bash

   $ cylc help       # Top level command help.
   $ cylc run --help # Example command-specific help.

Command help transcripts are printed in :ref:`CommandReference` and are
available from the GUI Help menu.

Cylc is *scriptable* - the error status returned by commands can be
relied on.

Graphical User Interface (GUI)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The cylc GUI covers the same functionality as the CLI, but it has more
sophisticated suite monitoring capability. It can start and stop suites, or
connect to suites that are already running; in either case, shutting down the
GUI does not affect the suite itself.

.. code-block:: bash

   $ gcylc & # or:
   $ cylc gui & # Single suite control GUI.
   $ cylc gscan & # Multi-suite monitor GUI.

Clicking on a suite in gscan, shown in :numref:`fig-gscan`, opens a
gcylc instance for it.

Suite Configuration
-------------------

Cylc suites are defined by extended-INI format ``suite.rc``
files (the main file format extension is section nesting). These reside
in *suite configuration directories* that may also contain a
``bin`` directory and any other suite-related files.

- For more on the suite configuration file format, see :ref:`SuiteDefinition`
  and :ref:`SuiteRCReference`.

Suite Registration
------------------

Suite registration creates a run directory (under ``~/cylc-run/`` by
default) and populates it with authentication files and a symbolic link to a
suite configuration directory. Cylc commands that parse suites can take
the file path or the suite name as input. Commands that interact with running
suites have to target the suite by name.

.. code-block:: bash

   # Target a suite by file path:
   $ cylc validate /path/to/my/suite/suite.rc
   $ cylc graph /path/to/my/suite/suite.rc

   # Register a suite:
   $ cylc register my.suite /path/to/my/suite/

   # Target a suite by name:
   $ cylc graph my.suite
   $ cylc validate my.suite
   $ cylc run my.suite
   $ cylc stop my.suite
   # etc.


.. _tutPassphrases:

Suite Passphrases
-----------------

Registration (above) also generates a suite-specific passphrase file under
``.service/`` in the suite run directory. It is loaded by the suite
server program at start-up and used to authenticate connections from client
programs.

Possession of a suite's passphrase file gives full control over it.
Without it, the information available to a client is determined by the suite's
public access privilege level.

For more on connection authentication, suite passphrases, and public access,
see :ref:`ConnectionAuthentication`.


.. _ImportTheExampleSuites:

Import The Example Suites
-------------------------

Run the following command to copy cylc's example suites and register them for
your own use:

.. code-block:: bash

   $ cylc import-examples /tmp


Rename The Imported Tutorial Suites
-----------------------------------

Suites can be renamed by simply renaming (i.e. moving) their run directories.
Make the tutorial suite names shorter, and print their locations with
``cylc print``:

.. code-block:: bash

   $ mv ~/cylc-run/examples/$(cylc --version)/tutorial ~/cylc-run/tut
   $ cylc print -ya tut
   tut/oneoff/jinja2  | /tmp/cylc-examples/7.0.0/tutorial/oneoff/jinja2
   tut/cycling/two    | /tmp/cylc-examples/7.0.0/tutorial/cycling/two
   tut/cycling/three  | /tmp/cylc-examples/7.0.0/tutorial/cycling/three
   # ...

See ``cylc print --help`` for other display options.

Suite Validation
----------------

Suite configurations can be validated to detect syntax (and other) errors:

.. code-block:: bash

   # pass:
   $ cylc validate tut/oneoff/basic
   Valid for cylc-6.0.0
   $ echo $?
   0
   # fail:
   $ cylc validate my/bad/suite
   Illegal item: [scheduling]special tusks
   $ echo $?
   1


Hello World in Cylc
-------------------

**suite**: ``tut/oneoff/basic``

Here's the traditional *Hello World* program rendered as a cylc
suite:

.. literalinclude:: ../../etc/examples/tutorial/oneoff/basic/suite.rc
   :language: cylc

Cylc suites feature a clean separation of scheduling configuration,
which determines *when* tasks are ready to run; and runtime
configuration, which determines *what* to run (and *where* and
*how* to run it) when a task is ready. In this example the
``[scheduling]`` section defines a single task called
``hello`` that triggers immediately when the suite starts
up. When the task finishes the suite shuts down. That this is a
*dependency graph* will be more obvious when more tasks are added.
Under the ``[runtime]`` section the
``script`` item defines a simple inlined
implementation for ``hello``: it sleeps for ten seconds,
then prints ``Hello World!``, and exits. This ends up in a *job script*
generated by cylc to encapsulate the task (below) and,
thanks to some defaults designed to allow quick
prototyping of new suites, it is submitted to run as a background job on
the suite host. In fact cylc even provides a default task implementation
that makes the entire ``[runtime]`` section technically optional:

.. literalinclude:: ../../etc/examples/tutorial/oneoff/minimal/suite.rc
   :language: cylc

(the resulting *dummy task* just prints out some identifying
information and exits).

Editing Suites
--------------

The text editor invoked by Cylc on suite configurations is determined
by cylc site and user global config files, as shown above in :ref:`CUI`.
Check that you have renamed the tutorial examples suites as described
just above and open the *Hello World* suite in your text editor:

.. code-block:: bash

   $ cylc edit tut/oneoff/basic # in-terminal
   $ cylc edit -g tut/oneoff/basic & # or GUI

Alternatively, start gcylc on the suite:

.. code-block:: bash

   $ gcylc tut/oneoff/basic &

and choose *Suite* ``->`` *Edit* from the menu.

The editor will be invoked from within the suite configuration directory
for easy access to other suite files (in this case there are none). There are
syntax highlighting control files for several text editors under
``<cylc-dir>/etc/syntax/``; see in-file comments for installation
instructions.


.. _RunningSuitesCLI:

Running Suites
--------------

CLI
^^^

Run ``tut/oneoff/basic`` using the ``cylc run`` command.
As a suite runs detailed timestamped information is written to a *suite log*
and progress can be followed with cylc's suite monitoring tools (below).
By default a suite server program daemonizes after printing a short message so
that you can exit the terminal or even log out without killing the suite:

.. code-block:: bash

   $ cylc run tut/oneoff/basic
               ._.
               | |                 The Cylc Suite Engine [7.0.0]
   ._____._. ._| |_____.           Copyright (C) 2008-2019 NIWA & British Crown (Met Office) & Contributors.
   | .___| | | | | .___|  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   | !___| !_! | | !___.  This program comes with ABSOLUTELY NO WARRANTY;
   !_____!___. |_!_____!  see `cylc warranty`.  It is free software, you
         .___! |           are welcome to redistribute it under certain
         !_____!                conditions; see `cylc conditions`.

   *** listening on https://nwp-1:43027/ ***

   To view suite server program contact information:
    $ cylc get-suite-contact tut/oneoff/basic

   Other ways to see if the suite is still running:
    $ cylc scan -n '\btut/oneoff/basic\b' nwp-1
    $ cylc ping -v --host=nwp-1 tut/oneoff/basic
    $ ps h -opid,args 123456  # on nwp-1


If you're quick enough (this example only takes 10-15 seconds to run) the
``cylc scan`` command will detect the running suite:

.. code-block:: bash

   $ cylc scan
   tut/oneoff/basic oliverh@nwp-1:43027

.. note::

   You can use the ``--no-detach`` and ``--debug`` options
   to ``cylc-run`` to prevent the suite from daemonizing (i.e. to make
   it stay attached to your terminal until it exits).

When a task is ready cylc generates a *job script* to run it, by
default as a background jobs on the suite host.  The job process ID is
captured, and job output is directed to log files in standard
locations under the suite run directory.

Log file locations relative to the suite run directory look like
``job/1/hello/01/`` where the first digit is the *cycle point* of
the task ``hello`` (for non-cycling tasks this is just ``1``); and the
final ``01`` is the *submit number* (so that job logs do not get
overwritten if a job is resubmitted for any reason).

The suite shuts down automatically once all tasks have succeeded.

GUI
^^^

The cylc GUI can start and stop suites, or (re)connect to suites that
are already running:

.. code-block:: bash

   $ cylc gui tut/oneoff/basic &

Use the tool bar *Play* button, or the *Control* ``->`` *Run* menu item, to
run the suite again. You may want to alter the suite configuration slightly
to make the task take longer to run. Try right-clicking on the
``hello`` task to view its output logs. The relative merits of the three
*suite views* - dot, text, and graph - will be more apparent later when we
have more tasks. Closing the GUI does not affect the suite itself.


.. _RemoteSuites:

Remote Suites
-------------

Suites can run on *localhost* or on a *remote* host.

To start up a suite on a given host, specify it explicitly via the
``--host=`` option to a ``run`` or ``restart`` command.

Otherwise, Cylc selects the best host to start up on from allowed
``run hosts`` as specified in the global config under
``[suite servers]``, which defaults to localhost. Should there be
more than one allowed host set, the *most suitable* is determined
according to the settings specified under ``[[run host select]]``,
namely exclusion of hosts not meeting suitability *thresholds*, if
provided, then ranking according to the given *rank* method.

Discovering Running Suites
--------------------------

Suites that are currently running can be detected with command line or
GUI tools:

.. code-block:: bash

   # list currently running suites and their port numbers:
   $ cylc scan
   tut/oneoff/basic oliverh@nwp-1:43001

   # GUI summary view of running suites:
   $ cylc gscan &

The scan GUI is shown in :numref:`fig-gscan`; clicking on a suite in
it opens gcylc.


.. _TaskIdentifiers:

Task Identifiers
----------------

At run time, task instances are identified by their *name* (see
:ref:`TaskNames`), which is determined entirely by the suite configuration,
and a *cycle point* which is usually a date-time or an integer:

.. code-block:: bash

   foo.20100808T00Z   # a task with a date-time cycle point
   bar.1              # a task with an integer cycle point (could be non-cycling)

Non-cycling tasks usually just have the cycle point ``1``, but this
still has to be used to target the task instance with cylc commands.

Job Submission: How Tasks Are Executed
--------------------------------------

**suite**: ``tut/oneoff/jobsub``

Task *job scripts* are generated by cylc to wrap the task implementation
specified in the suite configuration (environment, script, etc.) in
error trapping code, messaging calls to report task progress back to the suite
server program, and so forth. Job scripts are written to the *suite job log
directory* where they can be viewed alongside the job output logs. They
can be accessed at run time by right-clicking on the task in the cylc GUI, or
printed to the terminal:

.. code-block:: bash

   $ cylc cat-log tut/oneoff/basic hello.1


This command can also print the suite log (and stdout and stderr for suites
in daemon mode) and task stdout and stderr logs (see
``cylc cat-log --help``).

A new job script can also be generated on the fly for inspection:

.. code-block:: bash

   $ cylc jobscript tut/oneoff/basic hello.1

Take a look at the job script generated for ``hello.1`` during
the suite run above. The custom scripting should be clearly visible
toward the bottom of the file.

The ``hello`` task in the first tutorial suite defaults to
running as a background job on the suite host. To submit it to the Unix
``at`` scheduler instead, configure its job submission settings
as in ``tut/oneoff/jobsub``:

.. code-block:: cylc

   [runtime]
       [[hello]]
           script = "sleep 10; echo Hello World!"
           [[[job]]]
               batch system = at

Run the suite again after checking that ``at`` is running on your
system.

Cylc supports a number of different batch systems. Tasks
submitted to external batch queuing systems like ``at``,
``PBS``, ``SLURM``, ``Moab``, or ``LoadLeveler``, are displayed as
*submitted* in the cylc GUI until they start executing.

- For more on task job scripts, see :ref:`JobScripts`.
- For more on batch systems, see :ref:`AvailableMethods`.


Locating Suite And Task Output
------------------------------

If the ``--no-detach`` option is not used, suite stdout and
stderr will be directed to the suite run directory along with the
time-stamped suite log file, and task job scripts and job logs
(task stdout and stderr). The default suite run directory location is
``$HOME/cylc-run``:

.. code-block:: bash

   $ tree $HOME/cylc-run/tut/oneoff/basic/
   |-- .service              # location of run time service files
   |    |-- contact          # detail on how to contact the running suite
   |    |-- db               # private suite run database
   |    |-- passphrase       # passphrase for client authentication
   |    |-- source           # symbolic link to source directory
   |    |-- ssl.cert         # SSL certificate for the suite server
   |    `-- ssl.pem          # SSL private key
   |-- cylc-suite.db         # back compat symlink to public suite run database
   |-- share                 # suite share directory (not used in this example)
   |-- work                  # task work space (sub-dirs are deleted if not used)
   |    `-- 1                   # task cycle point directory (or 1)
   |        `-- hello              # task work directory (deleted if not used)
   |-- log                   # suite log directory
   |   |-- db                   # public suite run database
   |   |-- job                  # task job log directory
   |   |   `-- 1                   # task cycle point directory (or 1)
   |   |       `-- hello              # task name
   |   |           |-- 01                # task submission number
   |   |           |   |-- job              # task job script
   |   |           |   `-- job-activity.log # task job activity log
   |   |           |   |-- job.err          # task stderr log
   |   |           |   |-- job.out          # task stdout log
   |   |           |   `-- job.status       # task status file
   |   |           `-- NN -> 01          # symlink to latest submission number
   |   `-- suite                # suite server log directory
   |       |-- err                 # suite server stderr log (daemon mode only)
   |       |-- out                 # suite server stdout log (daemon mode only)
   |       `-- log                 # suite server event log (timestamped info)

The suite run database files, suite environment file,
and task status files are used internally by cylc. Tasks execute in
private ``work/`` directories that are deleted automatically
if empty when the task finishes. The suite
``share/`` directory is made available to all tasks (by
``$CYLC_SUITE_SHARE_DIR``) as a common share space. The task submission
number increments from ``1`` if a task retries; this is used as a
sub-directory of the log tree to avoid overwriting log files from earlier
job submissions.

The top level run directory location can be changed in site and user
config files if necessary, and the suite share and work locations can be
configured separately because of the potentially larger disk space
requirement.

Task job logs can be viewed by right-clicking on tasks in the gcylc
GUI (so long as the task proxy is live in the suite), manually
accessed from the log directory (of course), or printed to the terminal
with the ``cylc cat-log`` command:

.. code-block:: bash

   # suite logs:
   $ cylc cat-log    tut/oneoff/basic           # suite event log
   $ cylc cat-log -o tut/oneoff/basic           # suite stdout log
   $ cylc cat-log -e tut/oneoff/basic           # suite stderr log
   # task logs:
   $ cylc cat-log    tut/oneoff/basic hello.1   # task job script
   $ cylc cat-log -o tut/oneoff/basic hello.1   # task stdout log
   $ cylc cat-log -e tut/oneoff/basic hello.1   # task stderr log

- For a web-based interface to suite and task logs (and much more),
  see *Rose* in :ref:`SuiteStorageEtc`.
- For more on environment variables supplied to tasks, such as
  ``$CYLC_SUITE_SHARE_DIR``, see :ref:`TaskExecutionEnvironment`.


.. _ViewingSuiteLogsCylcReview:

Viewing Suite Logs in a Web Browser: Cylc Review
------------------------------------------------

The Cylc Review web service displays suite job logs and other information in
web pages, as shown in :numref:`fig-review-screenshot`. It can run under a
WSGI server (e.g. Apache with ``mod_wsgi``) as a service for all
users, or as an ad hoc service under your own user account.

If a central Cylc Review service has been set up at your site (e.g. as
described in :ref:`ConfiguringCylcReviewApache`) the URL will typically be
something like ``http://<server>/cylc-review/``.

.. _fig-review-screenshot:

.. figure:: graphics/png/orig/cylc-review-screenshot.png
   :align: center

   Screenshot of a Cylc Review web page

Otherwise, to start an ad hoc Cylc Review service to view your own suite logs
(or those of others, if you have read access to them), run:

.. code-block:: none

   setsid cylc review start 0</dev/null 1>/dev/null 2>&1 &

The service should start at ``http://<server>:8080`` (the port number
can optionally be set on the command line). Service logs are written to
``~/.cylc/cylc-review*``. Run ``cylc review`` to view
status information, and ``cylc review stop`` to stop the service.


.. _RemoteTasks:

Remote Tasks
------------

**suite**: ``tut/oneoff/remote``

The ``hello`` task in the first two tutorial suites defaults to
running on the suite host :ref:`RemoteSuites`. To make it run on a different
host instead change its runtime configuration as in ``tut/oneoff/remote``:

.. code-block:: cylc

   [runtime]
       [[hello]]
           script = "sleep 10; echo Hello World!"
           [[[remote]]]
               host = server1.niwa.co.nz

In general, a *task remote* is a user account, other than the account
running the suite server program, where a task job is submitted to run. It can
be on the same machine running the suite or on another machine.

A task remote account must satisfy several requirements:

- Non-interactive ssh must be enabled from the account running the suite
  server program to the account for submitting (and managing) the remote
  task job.
- Network settings must allow communication *back* from the remote task
  job to the suite, either by network ports or ssh, unless the last-resort one
  way *task polling* communication method is used.
- Cylc must be installed and runnable on the task remote account. Other
  software dependencies like graphviz are not required there.
- Any files needed by a remote task must be installed on the task
  host. In this example there is nothing to install because the
  implementation of ``hello`` is inlined in the suite configuration
  and thus ends up entirely contained within the task job script.

If your username is different on the task host, you can add a ``User``
setting for the relevant host in your ``~/.ssh/config``.
If you are unable to do so, the ``[[[remote]]]`` section also supports an
``owner=username`` item.

If you configure a task account according to the requirements cylc will invoke
itself on the remote account (with a login shell by default) to create log
directories, transfer any essential service files, send the task job script
over, and submit it to run there by the configured batch system.

Remote task job logs are saved to the suite run directory on the task remote,
not on the account running the suite. They can be retrieved by right-clicking
on the task in the GUI, or to have cylc pull them back to the suite account
automatically do this:

.. code-block:: cylc

   [runtime]
       [[hello]]
           script = "sleep 10; echo Hello World!"
           [[[remote]]]
               host = server1.niwa.co.nz
               retrieve job logs = True

This suite will attempt to ``rsync`` job logs from the remote
host each time a task job completes.

Some batch systems have considerable delays between the time when the job
completes and when it writes the job logs in its normal location. If this is
the case, you can configure an initial delay and retry delays for job log
retrieval by setting some delays. E.g.:

.. code-block:: cylc

   [runtime]
       [[hello]]
           script = "sleep 10; echo Hello World!"
           [[[remote]]]
               host = server1.niwa.co.nz
               retrieve job logs = True
               # Retry after 10 seconds, 1 minute and 3 minutes
               retrieve job logs retry delays = PT10S, PT1M, PT3M


Finally, if the disk space of the suite host is limited, you may want to set
``[[[remote]]]retrieve job logs max size=SIZE``. The value of SIZE can
be anything that is accepted by the ``--max-size=SIZE`` option of the
``rsync`` command. E.g.:

.. code-block:: cylc

   [runtime]
       [[hello]]
           script = "sleep 10; echo Hello World!"
           [[[remote]]]
               host = server1.niwa.co.nz
               retrieve job logs = True
               # Don't get anything bigger than 10MB
               retrieve job logs max size = 10M

It is worth noting that cylc uses the existence of a job's ``job.out``
or ``job.err`` in the local file system to indicate a successful job
log retrieval. If ``retrieve job logs max size=SIZE`` is set and both
``job.out`` and ``job.err`` are bigger than ``SIZE``
then cylc will consider the retrieval as failed. If retry delays are specified,
this will trigger some useless (but harmless) retries. If this occurs
regularly, you should try the following:

- Reduce the verbosity of STDOUT or STDERR from the task.
- Redirect the verbosity from STDOUT or STDERR to an alternate log file.
- Adjust the size limit with tolerance to the expected size of STDOUT or
  STDERR.

- For more on remote tasks see :ref:`RunningTasksOnARemoteHost`
- For more on task communications, see :ref:`TaskComms`.
- For more on suite passphrases and authentication,
  see :ref:`tutPassphrases` and :ref:`ConnectionAuthentication`.


Task Triggering
---------------

**suite**: ``tut/oneoff/goodbye``

To make a second task called ``goodbye`` trigger after
``hello`` finishes successfully, return to the original
example, ``tut/oneoff/basic``, and change the suite graph
as in ``tut/oneoff/goodbye``:

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           graph = "hello => goodbye"

or to trigger it at the same time as ``hello``,

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           graph = "hello & goodbye"

and configure the new task's behaviour under ``[runtime]``:

.. code-block:: cylc

   [runtime]
       [[goodbye]]
           script = "sleep 10; echo Goodbye World!"

Run ``tut/oneoff/goodbye`` and check the output from the new task:

.. code-block:: bash

   $ cat ~/cylc-run/tut/oneoff/goodbye/log/job/1/goodbye/01/job.out
     # or
   $ cylc cat-log -o tut/oneoff/goodbye goodbye.1
   JOB SCRIPT STARTING
   cylc (scheduler - 2014-08-14T15:09:30+12): goodbye.1 started at 2014-08-14T15:09:30+12
   cylc Suite and Task Identity:
     Suite Name  : tut/oneoff/goodbye
     Suite Host  : oliverh-34403dl.niwa.local
     Suite Port  : 43001
     Suite Owner : oliverh
     Task ID     : goodbye.1
     Task Host   : nwp-1
     Task Owner  : oliverh
     Task Try No.: 1

   Goodbye World!
   cylc (scheduler - 2014-08-14T15:09:40+12): goodbye.1 succeeded at 2014-08-14T15:09:40+12
   JOB SCRIPT EXITING (TASK SUCCEEDED)


Task Failure And Suicide Triggering
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**suite**: ``tut/oneoff/suicide``

Task names in the graph string can be qualified with a state indicator
to trigger off task states other than success:

.. code-block:: cylc

       graph = """
   a => b        # trigger b if a succeeds
   c:submit => d # trigger d if c submits
   e:finish => f # trigger f if e succeeds or fails
   g:start  => h # trigger h if g starts executing
   i:fail   => j # trigger j if i fails
               """

A common use of this is to automate recovery from known modes of failure:

.. code-block:: cylc

   graph = "goodbye:fail => really_goodbye"

i.e. if task ``goodbye`` fails, trigger another task that
(presumably) really says goodbye.

Failure triggering generally requires use of *suicide triggers* as
well, to remove the recovery task if it isn't required (otherwise it
would hang about indefinitely in the waiting state):

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           graph = """hello => goodbye
               goodbye:fail => really_goodbye
            goodbye => !really_goodbye # suicide"""

This means if ``goodbye`` fails, trigger
``really_goodbye``; and otherwise, if ``goodbye``
succeeds, remove ``really_goodbye`` from the suite.

Try running ``tut/oneoff/suicide``, which also configures
the ``hello`` task's runtime to make it fail, to see how this works.

- For more on suite dependency graphs see :ref:`ConfiguringScheduling`.
- For more on task triggering see :ref:`TriggerTypes`.


Runtime Inheritance
-------------------

**suite**: ``tut/oneoff/inherit``

The ``[runtime]`` section is actually a *multiple inheritance* hierarchy.
Each subsection is a *namespace* that represents a task, or if it is
inherited by other namespaces, a *family*. This allows common configuration
to be factored out of related tasks very efficiently.

.. literalinclude:: ../../etc/examples/tutorial/oneoff/inherit/suite.rc
   :language: cylc

The ``[root]`` namespace provides defaults for all tasks in the suite.
Here both tasks inherit ``script`` from ``root``, which they
customize with different values of the environment variable
``$GREETING``. 

.. note::

   Inheritance from ``root`` is
   implicit; from other parents an explicit ``inherit = PARENT``
   is required, as shown below.

- For more on runtime inheritance, see :ref:`NIORP`.

Triggering Families
-------------------

**suite**: ``tut/oneoff/ftrigger1``

Task families defined by runtime inheritance can also be used as
shorthand in graph trigger expressions. To see this, consider two
"greeter" tasks that trigger off another task ``foo``:

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           graph = "foo => greeter_1 & greeter_2"

If we put the common greeting functionality of ``greeter_1``
and ``greeter_2`` into a special ``GREETERS`` family,
the graph can be expressed more efficiently like this:

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           graph = "foo => GREETERS"

i.e. if ``foo`` succeeds, trigger all members of
``GREETERS`` at once. Here's the full suite with runtime
hierarchy shown:

.. literalinclude:: ../../etc/examples/tutorial/oneoff/ftrigger1/suite.rc
   :language: cylc

.. note::

   We recommend given ALL-CAPS names to task families to help
   distinguish them from task names. However, this is just a convention.

Experiment with the ``tut/oneoff/ftrigger1`` suite to see
how this works.

Triggering Off Of Families
--------------------------

**suite**: ``tut/oneoff/ftrigger2``

Tasks (or families) can also trigger *off* other families, but
in this case we need to specify what the trigger means in terms of
the upstream family members. Here's how to trigger another task
``bar`` if all members of ``GREETERS`` succeed:

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           graph = """foo => GREETERS
               GREETERS:succeed-all => bar"""

Verbose validation in this case reports:

.. code-block:: bash

   $ cylc val -v tut/oneoff/ftrigger2
   ...
   Graph line substitutions occurred:
     IN: GREETERS:succeed-all => bar
     OUT: greeter_1:succeed & greeter_2:succeed => bar
   ...

Cylc ignores family member qualifiers like ``succeed-all`` on
the right side of a trigger arrow, where they don't make sense, to
allow the two graph lines above to be combined in simple cases:

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           graph = "foo => GREETERS:succeed-all => bar"

Any task triggering status qualified by ``-all`` or
``-any``, for the members, can be used with a family trigger.
For example, here's how to trigger ``bar`` if all members
of ``GREETERS`` finish (succeed or fail) and any of them succeed:

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           graph = """foo => GREETERS
       GREETERS:finish-all & GREETERS:succeed-any => bar"""

(use of ``GREETERS:succeed-any`` by itself here would trigger
``bar`` as soon as any one member of ``GREETERS``
completed successfully). Verbose validation now begins to show how
family triggers can simplify complex graphs, even for this tiny
two-member family:

.. code-block:: bash

   $ cylc val -v tut/oneoff/ftrigger2
   ...
   Graph line substitutions occurred:
     IN: GREETERS:finish-all & GREETERS:succeed-any => bar
     OUT: ( greeter_1:succeed | greeter_1:fail ) & \
          ( greeter_2:succeed | greeter_2:fail ) & \
          ( greeter_1:succeed | greeter_2:succeed ) => bar
   ...

Experiment with ``tut/oneoff/ftrigger2`` to see how this works.

- For more on family triggering, see :ref:`FamilyTriggers`.


Suite Visualization
-------------------

You can style dependency graphs with an optional
``[visualization]`` section, as shown in ``tut/oneoff/ftrigger2``:

.. code-block:: cylc

   [visualization]
       default node attributes = "style=filled"
       [[node attributes]]
           foo = "fillcolor=#6789ab", "color=magenta"
           GREETERS = "fillcolor=#ba9876"
           bar = "fillcolor=#89ab67"

To display the graph in an interactive viewer:

.. code-block:: bash

   $ cylc graph tut/oneoff/ftrigger2 &    # dependency graph
   $ cylc graph -n tut/oneoff/ftrigger2 & # runtime inheritance graph

It should look like :numref:`fig-tut-hello-multi` (with the
``GREETERS`` family node expanded on the right).

.. todo::
   Create sub-figures if possible: for now hacked as separate figures with
   link to first, and caption on final, displayed figure.

.. _fig-tut-hello-multi:

.. figure:: graphics/png/orig/tut-hello-multi-1.png
   :align: center

.. figure:: graphics/png/orig/tut-hello-multi-2.png
   :align: center

.. figure:: graphics/png/orig/tut-hello-multi-3.png
   :align: center

   The ``tut/oneoff/ftrigger2`` dependency and runtime inheritance graphs


Graph styling can be applied to entire families at once, and custom
"node groups" can also be defined for non-family groups.


External Task Scripts
---------------------

**suite**: ``tut/oneoff/external``

The tasks in our examples so far have all had inlined implementation, in
the suite configuration, but real tasks often need to call external
commands, scripts, or executables. To try this, let's return to the
basic Hello World suite and cut the implementation of the task
``hello`` out to a file ``hello.sh`` in the suite bin directory:

.. literalinclude:: ../../etc/examples/tutorial/oneoff/external/bin/hello.sh
   :language: bash

Make the task script executable, and change the ``hello`` task
runtime section to invoke it:

.. literalinclude:: ../../etc/examples/tutorial/oneoff/external/suite.rc
   :language: cylc

If you run the suite now the new greeting from the external task script
should appear in the ``hello`` task stdout log. This works
because cylc automatically adds the suite bin directory to
``$PATH`` in the environment passed to tasks via their job
scripts. To execute scripts (etc.) located elsewhere you can
refer to the file by its full file path, or set ``$PATH``
appropriately yourself (this could be done via
``$HOME/.profile``, which is sourced at the top of the task job
script, or in the suite configuration itself).

.. note::

   The use of ``set -e`` above to make the script abort on
   error. This allows the error trapping code in the task job script to
   automatically detect unforeseen errors.

Cycling Tasks
-------------

**suite**: ``tut/cycling/one``

So far we've considered non-cycling tasks, which finish without spawning
a successor.

Cycling is based around iterating through date-time or integer sequences. A
cycling task may run at each cycle point in a given sequence (cycle). For
example, a sequence might be a set of date-times every 6 hours starting from a
particular date-time. A cycling task may run for each date-time item (cycle
point) in that sequence.

There may be multiple instances of this type of task running in parallel, if
the opportunity arises and their dependencies allow it. Alternatively, a
sequence can be defined with only one valid cycle point - in that case, a task
belonging to that sequence may only run once.

Open the ``tut/cycling/one`` suite:

.. literalinclude:: ../../etc/examples/tutorial/cycling/one/suite.rc
   :language: cylc

The difference between cycling and non-cycling suites is all in the
``[scheduling]`` section, so we will leave the
``[runtime]`` section alone for now (this will result in
cycling dummy tasks).

.. note::

   The graph is now defined under a new section heading that makes each
   task under it have a succession of cycle points ending in ``00`` or
   ``12`` hours, between specified initial and final cycle
   points (or indefinitely if no final cycle point is given), as shown in
   :numref:`fig-tut-one`.

.. todo::
   Image out of date now.

.. _fig-tut-one:

.. figure:: graphics/png/orig/tut-one.png
   :align: center

   The ``tut/cycling/one`` suite

If you run this suite instances of ``foo`` will spawn in parallel out
to the *runahead limit*, and each ``bar`` will trigger off the
corresponding instance of ``foo`` at the same cycle point. The
runahead limit, which defaults to a few cycles but is configurable, prevents
uncontrolled spawning of cycling tasks in suites that are not constrained by
clock triggers in real time operation.

Experiment with ``tut/cycling/one`` to see how cycling tasks work.

ISO 8601 Date-Time Syntax
^^^^^^^^^^^^^^^^^^^^^^^^^

The suite above is a very simple example of a cycling date-time workflow. More
generally, cylc comprehensively supports the ISO 8601 standard for date-time
instants, intervals, and sequences. Cycling graph sections can be specified
using full ISO 8601 recurrence expressions, but these may be simplified
by assuming context information from the suite - namely initial and final cycle
points. One form of the recurrence syntax looks like
``Rn/start-date-time/period`` (``Rn`` means run ``n`` times). In the example
above, if the initial cycle point
is always at ``00`` or ``12`` hours then ``[[[T00,T12]]]`` could be
written as ``[[[PT12H]]]``, which is short for
``[[[R/initial-cycle-point/PT12H/]]]`` - i.e. run every 12 hours
indefinitely starting at the initial cycle point. It is possible to add
constraints to the suite to only allow initial cycle points at ``00`` or
``12`` hours e.g.

.. code-block:: cylc

   [scheduling]
       initial cycle point = 20130808T00
       initial cycle point constraints = T00, T12

.. todo::
   Runahead factor now.

- For a comprehensive description of ISO 8601 based date-time cycling,
  see :ref:`AdvancedCycling`
- For more on runahead limiting in cycling suites,
  see :ref:`RunaheadLimit`.


.. _TutInterCyclePointTriggers:

Inter-Cycle Triggers
^^^^^^^^^^^^^^^^^^^^

**suite**: ``tut/cycling/two``

The ``tut/cycling/two`` suite adds inter-cycle dependence
to the previous example:

.. code-block:: cylc

   [scheduling]
       [[dependencies]]
           # Repeat with cycle points of 00 and 12 hours every day:
           [[[T00,T12]]]
               graph = "foo[-PT12H] => foo => bar"

For any given cycle point in the sequence defined by the
cycling graph section heading, ``bar`` triggers off
``foo`` as before, but now ``foo`` triggers off its own
previous instance ``foo[-PT12H]``. Date-time offsets in
inter-cycle triggers are expressed as ISO 8601 intervals (12 hours
in this case). :numref:`fig-tut-two` shows how this connects the
cycling graph sections together.

.. _fig-tut-two:

.. figure:: graphics/png/orig/tut-two.png
   :align: center

   The ``tut/cycling/two`` suite

Experiment with this suite to see how inter-cycle triggers work.

.. note::

   The first instance of ``foo``, at suite start-up, will
   trigger immediately in spite of its inter-cycle trigger, because cylc
   ignores dependence on points earlier than the initial cycle point.
   However, the presence of an inter-cycle trigger usually implies something
   special has to happen at start-up. If a model depends on its own previous
   instance for restart files, for example, then some special process has to
   generate the initial set of restart files when there is no previous cycle
   point to do it. The following section shows one way to handle this
   in cylc suites.


.. _initial-non-repeating-r1-tasks:

Initial Non-Repeating (R1) Tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**suite**: ``tut/cycling/three``

Sometimes we want to be able to run a task at the initial cycle point, but
refrain from running it in subsequent cycles. We can do this by writing an
extra set of dependencies that are only valid at a single date-time cycle
point. If we choose this to be the initial cycle point, these will only apply
at the very start of the suite.

The cylc syntax for writing this single date-time cycle point occurrence is
``R1``, which stands for ``R1/no-specified-date-time/no-specified-period``.
This is an adaptation of part of the ISO 8601 date-time standard's recurrence
syntax (``Rn/date-time/period``) with some special context information
supplied by cylc for the ``no-specified-*`` data.

The ``1`` in the ``R1`` means run once. As we've specified
no date-time, Cylc will use the initial cycle point date-time by default,
which is what we want. We've also missed out specifying the period - this is
set by cylc to a zero amount of time in this case (as it never
repeats, this is not significant).

For example, in ``tut/cycling/three``:

.. code-block:: cylc

   [cylc]
       cycle point time zone = +13
   [scheduling]
       initial cycle point = 20130808T00
       final cycle point = 20130812T00
       [[dependencies]]
           [[[R1]]]
               graph = "prep => foo"
           [[[T00,T12]]]
               graph = "foo[-PT12H] => foo => bar"

This is shown in :numref:`fig-tut-three`.

.. note::

   The time zone has been set to ``+1300`` in this case,
   instead of UTC (``Z``) as before. If no time zone or UTC mode was set,
   the local time zone of your machine will be used in the cycle points.

At the initial cycle point, ``foo`` will depend on ``foo[-PT12H]`` and also
on ``prep``:

.. code-block:: cylc

   prep.20130808T0000+13 & foo.20130807T1200+13 => foo.20130808T0000+13

Thereafter, it will just look like e.g.:

.. code-block:: cylc

   foo.20130808T0000+13 => foo.20130808T1200+13

However, in our initial cycle point example, the dependence on
``foo.20130807T1200+13`` will be ignored, because that task's cycle
point is earlier than the suite's initial cycle point and so it cannot run.
This means that the initial cycle point dependencies for ``foo``
actually look like:

.. code-block:: cylc

   prep.20130808T0000+13 => foo.20130808T0000+13


.. _fig-tut-three:

.. figure:: graphics/png/orig/tut-three.png
   :align: center

   The ``tut/cycling/three`` suite

- ``R1`` tasks can also be used to make something special
  happen at suite shutdown, or at any single cycle point throughout the
  suite run. For a full primer on cycling syntax, see :ref:`AdvancedCycling`.


.. _TutInteger:

Integer Cycling
^^^^^^^^^^^^^^^

**suite**: ``tut/cycling/integer``

Cylc can do also do integer cycling for repeating workflows that are not
date-time based.

Open the ``tut/cycling/integer`` suite, which is plotted in
:numref:`fig-tut-int`.

.. literalinclude:: ../../etc/examples/tutorial/cycling/integer/suite.rc
   :language: cylc

.. _fig-tut-int:

.. figure:: graphics/png/orig/tut-cyc-int.png
   :align: center

   The ``tut/cycling/integer`` suite

The integer cycling notation is intended to look similar to the ISO 8601
date-time notation, but it is simpler for obvious reasons. The example suite
illustrates two recurrence forms,
``Rn/start-point/period`` and
``Rn/period/stop-point``, simplified somewhat using suite context
information (namely the initial and final cycle points). The first form is
used to run one special task called ``start`` at start-up, and for the
main cycling body of the suite; and the second form to run another special task
called ``stop`` in the final two cycles. The ``P`` character
denotes period (interval) just like in the date-time notation.
``R/1/P2`` would generate the sequence of points ``1,3,5,...``.

- For more on integer cycling, including a more realistic usage example
  see :ref:`IntegerCycling`.


Jinja2
------

**suite**: ``tut/oneoff/jinja2``

Cylc has built in support for the Jinja2 template processor, which
allows us to embed code in suite configurations to generate the
final result seen by cylc.

The ``tut/oneoff/jinja2`` suite illustrates two common
uses of Jinja2: changing suite content or structure based on the value
of a logical switch; and iteratively generating dependencies and runtime
configuration for groups of related tasks:

.. literalinclude:: ../../etc/examples/tutorial/oneoff/jinja2/suite.rc
   :language: cylc

To view the result of Jinja2 processing with the Jinja2 flag
``MULTI`` set to ``False``:

.. code-block:: bash

   $ cylc view --jinja2 --stdout tut/oneoff/jinja2

.. code-block:: cylc

   [meta]
       title = "A Jinja2 Hello World! suite"
   [scheduling]
       [[dependencies]]
           graph = "hello"
   [runtime]
       [[hello]]
           script = "sleep 10; echo Hello World!"

And with ``MULTI`` set to ``True``:

.. code-block:: bash

   $ cylc view --jinja2 --stdout tut/oneoff/jinja2

.. code-block:: cylc

   [meta]
       title = "A Jinja2 Hello World! suite"
   [scheduling]
       [[dependencies]]
           graph = "hello => BYE"
   [runtime]
       [[hello]]
           script = "sleep 10; echo Hello World!"
       [[BYE]]
           script = "sleep 10; echo Goodbye World!"
       [[ goodbye_0 ]]
           inherit = BYE
       [[ goodbye_1 ]]
           inherit = BYE
       [[ goodbye_2 ]]
           inherit = BYE


Task Retry On Failure
---------------------

**suite**: ``tut/oneoff/retry``

Tasks can be configured to retry a number of times if they fail.
An environment variable ``$CYLC_TASK_TRY_NUMBER`` increments
from ``1`` on each successive try, and is passed to the task to allow
different behaviour on the retry:

.. literalinclude:: ../../etc/examples/tutorial/oneoff/retry/suite.rc
   :language: cylc

If a task with configured retries fails, it goes into the *retrying* state
until the next retry delay is up, then it resubmits. It only enters the
*failed* state on a final definitive failure.

If a task with configured retries is *killed* (by ``cylc kill`` or
via the GUI) it goes to the *held* state so that the operator can decide
whether to release it and continue the retry sequence or to abort the retry
sequence by manually resetting it to the *failed* state.

Experiment with ``tut/oneoff/retry`` to see how this works.

Other Users' Suites
-------------------

If you have read access to another user's account (even on another host)
it is possible to use ``cylc monitor`` to look at their suite's
progress without full shell access to their account. To do this, you
will need to copy their suite passphrase to

.. code-block:: bash

   $HOME/.cylc/SUITE_OWNER@SUITE_HOST/SUITE_NAME/passphrase

(use of the host and owner names is optional here - see :ref:`passphrases`)
*and* also retrieve the port number of the running suite from:

.. code-block:: bash

   ~SUITE_OWNER/cylc-run/SUITE_NAME/.service/contact

Once you have this information, you can run

.. code-block:: bash

   $ cylc monitor --user=SUITE_OWNER --port=SUITE_PORT SUITE_NAME

to view the progress of their suite.

Other suite-connecting commands work in the same way; see
:ref:`RemoteControl`.

Other Things To Try
-------------------

Almost every feature of cylc can be tested quickly and easily with a
simple dummy suite. You can write your own, or start from one of the
example suites in ``/path/to/cylc/examples`` (see use of
``cylc import-examples`` above) - they all run "out the box"
and can be copied and modified at will.

- Change the suite runahead limit in a cycling suite.
- Stop a suite mid-run with ``cylc stop``, and restart
  it again with ``cylc restart``.
- Hold (pause) a suite mid-run with ``cylc hold``,
  then modify the suite configuration and ``cylc reload`` it
  before using ``cylc release`` to continue (you can also
  reload without holding).
- Use the gcylc View menu to show the task state color key and
  watch tasks in the ``task-states`` example evolve
  as the suite runs.
- Manually re-run a task that has already completed or failed,
  with ``cylc trigger``.
- Use an *internal queue* to prevent more than an alotted number
  of tasks from running at once even though they are ready -
  see :ref:`InternalQueues`.
- Configure task event hooks to send an email, or shut the suite down,
  on task failure.
