Author: FCM team

# FCM: User Guide: System Administration

## Introduction

This chapter provides an administration guide for managers of projects or systems which are using FCM.

Note that, where this section refers to the FCM team this applies only to Met Office users. Users at other sites will either need to refer to the equivalent team within their organisation or will need to perfom these tasks themselves.

## Subversion

### Repository Layout

In theory you can set up your repository to have any random layouts. However, many `fcm` commands have to make assumptions on a set of working practices in order to function. The easiest way is to go with the default:

    <root>
        |
        |-- <project 1>
        |       |
        |       |-- trunk
        |       |-- branches
        |       |-- tags
        |
        |-- <project 2>
        |       |
        |       |-- trunk
        |       |-- branches
        |       |-- tags
        |
        |-- ...

In the default layout, each project is a sub-directory under the repository root. Each project has a `trunk` sub-directory, and optionally a `branches` sub-directory and a `tags` sub-directory. The main line of development of the project lives directly under the `trunk` sub-directory. A branch lives 3 levels under the `branches` sub-directory. A tag lives 1 level under the `tags` sub-directory.

FCM allows you to customise the layout for each repository by adding an `fcm:layout` property at the HEAD of root of the repository. E.g.:

    (shell prompt)$ fcm co -q -N svn://host/repos repos-root
    (shell prompt)$ cd repos-root
    (shell prompt)$ fcm pe fcm:layout .

The default settings are given in the following. In the editor started by `fcm pe`, add these settings and modify the VALUE of each KEY=VALUE pair.

    depth-project =
    depth-branch = 3
    depth-tag = 1
    dir-trunk = trunk
    dir-branch = branches
    dir-tag = tags
    level-owner-branch = 2
    level-owner-tag =
    template-branch = {category}/{owner}/{name_prefix}{name}
    template-tag =

The settings will become effective when you `fcm commit` them. An empty VALUE denotes an undefined value. The meanings of the settings are described below:

`depth-project`
  Number of sub-directories expected to be used by the name of a project. An undefined value means that a project can live under an arbitrary number of sub-directories (or directly) below the repository root.
`depth-branch`
  Number of sub-directories (under the sub-directory defined by `dir-branch`) expected to be used by the name of a branch. This setting must be defined.
`depth-tag`
  Number of sub-directories (under the sub-directory defined by `dir-tag`) expected to be used by the name of a tag. This setting must be defined.
`dir-trunk`
  The sub-directory (under the project) where the trunk source tree lives. This setting must be defined.
`dir-branch`
  The sub-directory (under the project) where (the sub-directories containing) all the branch source trees live.
`dir-tag`
  The sub-directory (under the project) where (the sub-directories containing) all the tag source trees live.
`level-owner-branch`
  The sub-directory level in the name of a branch containing its owner.
`level-owner-tag`
  The sub-directory level in the name of a tag containing its owner.
`template-branch`
  The template string to construct a branch name.
`template-tag`
  The template string to construct a tag name.

You will need to decide whether to use a single project tree for your system or whether to use multiple projects.

Advantages of a single project tree:

- Changes to any part of the system can always be committed as a single logical changeset. If you split your system into multiple projects then you may have occasions when a logical change involves more than one project and hence requires multiple commits (and branches).

Disadvantages of a single project tree:

- If you have a large system then your working copies may become very large and unwieldy. Basic commands such as `checkout` and `status` can become frustratingly slow if your working copy is too large.
- Depending on how you work, you may end up doing lots more merges of files that are unrelated to your work.

One common approach is to split the admin type files (e.g. site configurations that are unrelated to the main release) into a separate project from the core system files. If you include any large data files under version control you may also want to use a separate project for them to avoid making your working copies very large when editing code.

Note that there is often no obvious right or wrong answer so you just have to make a decision and see how it works out. You can always re-arrange your repository in the future (although be aware that this will break any changes being prepared on branches at the time).

You also need to decide whether your system requires its own repository (or multiple repositories) or whether it can share with another system.

- The main disadvantage of having separate repositories for each system is the maintenance overhead (although this is almost all automated by the FCM team so is not a big deal).
- We normally configure a single Trac environment per repository. If the repository contains multiple systems then it makes it difficult to use the Trac milestones to handle system releases. However, Trac now supports restricting itself to a sub-directory within a repository so, again, this is not a big deal.
- If you share a repository with other systems then your revision numbers can increase even when there are no changes to your system. This doesn't matter but some people don't like it.

For simplicity, in most cases you will probably want your own repository for your system.

You will not normally want to have multiple repositories for a system. One exception may be if you are storing large data files where you might not want to keep all the old versions for ever. Removing old versions can't be done without changing all the revision numbers which would mess up all your code history and Trac tickets. Storing the large data files in a separate repository reduces the impact if you do decide to remove old versions in the future. One disadvantage of this approach is that, for the moment at least, Trac only handles one repository so you will need a separate Trac environment for the data files.

For further details please see the section Planning Your Repository Organization from the Subversion book.

### Creating a Repository

Normally the FCM team will help you to set up your initial repository. However, it is quite simple if you need to do it yourself. First you need to issue the command `svnadmin create /path/to/repos`. This creates an empty repository which is now ready to accept an initial import. To do so, you should create a directory tree in a suitable location, and issue the `fcm project-create` command. At the root of the repository should be the project directories. Each project should then contain the `trunk` sub-directory. The sub-directories `branches` and `tags` are optional. You can import your source files to the `trunk` after the project is created. For example, if your directory tree is located at `$HOME/foo`, you will do the following to import it to a new repository:

    (SHELL PROMPT)$ svnadmin create FOO_svn
    (SHELL PROMPT)$ fcm project-create FOO file://$PWD/FOO_svn
    (SHELL PROMPT)$ fcm checkout file://$PWD/FOO_svn/FOO $HOME/svn-wc/foo
    (SHELL PROMPT)$ cd $HOME/svn-wc/foo
    (SHELL PROMPT)$ cp -r $HOME/foo/* .
    (SHELL PROMPT)$ fcm add *
    (SHELL PROMPT)$ fcm status
    (SHELL PROMPT)$ fcm commit

Note that the `svnadmin` command takes a PATH as an argument, as opposed to a URL for the `svn` command.

For further details please see the section Planning Your Repository Organization from the Subversion book.

### Access Control

Restrictions such as preventing anonymous read access or restricting write access to the trunk to a limited set of users can be arranged if necessary.

### Repository Hosting

The FCM team will organise the hosting of your repository. A number of facilities will be set up for you as standard.

Your repository will be set up on a central FCM server and access will be provided via `svnserve` (which we use in preference to Apache for performance reasons). The FCM team will advise you of the URL, and put in place standard hook scripts and backup procedures.

Note that if you want to use a Subversion repository for your own individual use there is no need to get the FCM team to host it. You can simply create your repository and then use a `file://` URL to access it.

## Trac

### Trac Configuration

Normally the FCM team will set up your Trac environment for you. This section describes some things you may wish to be configured. This can be done when the Trac environment is set up or later if you are unsure what you will require at first.

#### Access Control

You will not normally want to allow anonymous users to make changes to your Trac environment since this means that changes may not get identified with a userid. The FCM team will normally set up your Trac environment such that any authenticated users can make changes. Further restrictions such as restricting write access to named accounts or preventing anonymous read access can be arranged if necessary.

The system manager will normally be given TRAC_ADMIN privileges. This allows them to do additional things which normal users cannot do such as:

- Delete wiki pages (the latest version or the entire page).
- Add or modify milestones, components and versions.
- Modify ticket descriptions and delete ticket attachments.
- Make wiki pages read-only.
- Alter the permissions.

For further details please see the section Trac Permissions from the Trac documentation.

#### Email Notification

By default, each Trac environment is configured such that the owner and reporter and anyone on the CC list are notified whenever a change is made to a ticket. If system mangers wish to be notified of all ticket changes then this can also be configured. Alternatively, email notifications can be disabled if they are not wanted.

#### Other Configurable Items

There are lots of other things that can be configured in your Trac environment such as:

- Custom fields
- System icon
- Stylesheets

For further details please see the sections The Trac Configuration File and The Trac Ticket System from the Trac documentation.

### Trac Hosting

The FCM team will organise the hosting of your Trac environment. It will be set up on the same server that hosts your Subversion repository and access will be provided via a web server. The FCM team will advise you of the URL, and put in place the backup procedures.

## FCM Keywords

When you set up a repository for a new project, you will normally want the FCM team to set up a URL keyword for it in the FCM central configuration file. The name of the project should be a short string containing only word characters.

Individual projects can store revision keywords using the Subversion property `fcm:revision` at registered URLs. Using the UM as an example: if `UM` is a registered URL keyword, you can add the `fcm:revision` property at the head of the UM project by doing a non-recursive checkout. E.g.:

    (prompt)$ fcm co -q -N fcm:um um
    (prompt)$ fcm pe fcm:revision um

In the editor, add the following and `fcm commit`:

    vn6.3 = 402
    vn6.4 = 1396
    vn6.5 = 2599
    vn6.6 = 4913
    vn7.0 = 6163

In a subsequent invocation of `fcm`, if a revision keyword is specified for a URL in the UM namespace, the command will attempt to load it from the `fcm:revision` property at the head of the UM project. Revision keywords can also be defined in the FCM central configuration file if you prefer.

If the project has an associated Trac browser, you can also declare browser URL mapping in the central configuration file. This allows FCM to associate the Subversion URL with a Trac browser URL. There is an automatic default for mapping URLs hosted by the FCM team at the Met Office. External users of FCM may want to adjust this default for their site.

To change the default browser URL mapping, you need to make some `browser.*[namespace] = value` declarations in your site's `$FCM/etc/fcm/keyword.cfg` file. There are 3 components to this declaration: browser.comp-pat, browser.loc-tmpl and browser.rev-tmpl. The browser.comp-pat is a regular expression, which is used to separate the scheme-specific part of a version control system URL into a number of components by capturing its substrings. These components are then used to fill in the numbered fields in the browser.loc-tmpl. The template should have one more field than the number of components captured by browser.comp-pat. The last field is used to place the revision, which is generated via the browser.rev-tmpl. This template should have a single numbered field for filling in the revision number. This is best demonstrated by an example. Consider the declarations:

    browser.comp-pat = (?msx-i:\A//([^/]+)/(.*)\z)
    browser.loc-tmpl = http://{1}/intertrac/source:{2}{3}
    browser.rev-tmpl = @{1}

If we have a Subversion URL `svn://repos/path/to/a/file`, the browser.comp-pat will capture the components [`repos`, `path/to/a/file`]. When this is applied to the `browser.loc-tmpl`, {1} will be translated to `repos` and {2} will be translated to `path/to/a/file`. A revision is not given in this case, and so {3} is inserted with an empty string. The result is `http://repos/intertrac/path/to/a/file`. If the revision is `1357`, the browser.rev-tmpl will be used to translate it to `@1357`, which is then inserted to {3} of the browser.loc-tmpl. The result is therefore `http://repos/intertrac/path/to/a/file@1357`.

For more information on how to set up the keywords, please refer to Repository & Revision Keywords and the Annex: FCM Configuration File.

## FCM Make Configuration

The `fcm make` command (for invoking the extract and build systems) is very flexibile and can be used in lots of different ways. It is therefore difficult to give specific advice explaining how to configure them. However, based on experience with a number of systems, the following general advice can be offered.

- Standard FCM Make configuration files should be defined and stored within the repository. Users then include these files into their configurations, before applying their local changes.
- The files should be designed to include one another in a hierarchy. For example, you may have one core file which defines all the repository and source locations plus a series of platform/compiler specific files which include the core file. More complex setups are also possible if you need to cater for other options such as different optimisation levels, 32/64 bit, etc.
- When including other configuration files, always make use of the special $HERE variable (rather than, for instance, referring to a fixed repository location). When your configuration file is parsed, this special variable is normally expanded into the container directory of the current configuration file. This means that the include statements should work correctly whether you are referring to configuration files in the repository trunk, in a branch or in a local working copy.
- Make good use of variables (e.g. `$name_spaces`) to simplify repetitive declarations and make your configuration files easier to maintain.
- Use continuation lines to split long lines and make them easier to read.

Probably the best advice is to look at what has already been set up for other systems. The FCM team can advise on the best systems to examine.

When you create a stable build you should keep a FCM Make configuration file that can reproduce the build. One easy way to do this is to create your build using the standard configuration files and the latest versions of the code. You can then save the configuration file which is created on success.

## Maintaining Alternate Versions of Namelists and Data Files

Sometimes it is useful to be able to access particular revisions of some directories from a FCM repository without having to go via Subversion. Typical examples are namelist or data files used as inputs to a program. The `fcm export-items` command is designed to help with this. It can be used to maintain a set of extracted version directories from a FCM repository. The command has the following options:

`--config-file=PATH`, `--file=PATH`, `-f PATH`
  Specifies the path to the configuration file. (default=`$PWD/fcm-export-items.cfg`)
`--directory=PATH`, `-C PATH`
  Specifies the path to the destination. (default=`$PWD`)
`--new`
  Specifies the new mode. In this mode, everything is re-exported. Otherwise, the system runs in incremental mode, in which the version directories are only updated if they do not already exist.

The 1st argument SOURCE should be the URL of a branch in a Subversion repository with the standard FCM layout.

The configuration file should be in the deprecated FCM 1 configuration format. The label in each entry should be a path relative to the source URL. If the path ends in `*` then the path is expanded recursively and any sub-directories containing regular files are added to the list of relative paths to extract. The value may be empty, or it may be a list of space separated conditions. Each condition is a conditional operator (`>`, `>=`, `<`, `<=`, `==` or `!=`) followed by a revision number. The command uses the revision log to determine the revisions at which the relative path has been updated in the source URL. If these revisions also satisfy the conditions set by the user, they will be considered in the export.

Example:

    (SHELL PROMPT)$ cat >fcm-export-items.cfg <<EOF
    namelists/VerNL_AreaDefinition   >1000 !=1234
    namelists/VerNL_GRIBToPPCode     >=600 <3000
    namelists/VerNL_StationList
    elements/*                       >1000
    EOF
    (SHELL PROMPT)$ fcm export-items fcm:ver_tr

N.B.

1. Each time a sub-directory is revised, the script assigns a sequential v number for the item. Each v number for a sub-directory, therefore, is associated with a revision number. For each exported revision directory, there is a corresponding v number symbolic link pointing to it.
2. The system also creates a symbolic link `latest` to point to the latest exported revision directory.

## Defining Working Practices and Policies

Some options on working practices and policies are defined in the chapter on Code Management Working Practices. Individual projects should document the approach they have adopted. In addition, each project may also need to define its own working practices and policies to suit its local need. For example each project may need to specify:

- Whether changes are allowed directly on the trunk or whether branches have to be used in all cases.
- Whether all users are allowed to make changes to the trunk.
- Whether Trac tickets have to be raised for all changes to the trunk.
- Whether Trac tickets should be raised for all support queries or whether a Trac ticket should only be raised once there is an agreed "issue".
- Whether branches should normally be made from the latest code or from a stable release.
- Whether a user is allowed to resolve conflicts directly when merging a branch into the trunk or whether he/she should merge the trunk into the branch and resolve the conflicts in the branch first.
- Whether all code changes to the trunk need to be reviewed.
- What testing is required before changes can be merged to the trunk.
- Whether history entries are maintained in source files or whether individual source files changes need to be described in the Subversion log message.
- Branch deletion policy.
- Whether any files in the project require locking before being changed.