
1 Overview

  WebJob downloads a program or script from a remote WebJob server
  and executes it in one unified operation.  Any output produced
  by the program/script is packaged up and sent to a remote, possibly
  different, WebJob server.  WebJob is useful because it provides
  a mechanism for running known good programs on damaged or potentially
  compromised systems.  This makes it ideal for remote diagnostics,
  incident response, and evidence collection.  WebJob also provides
  a framework that is conducive to centralized management.  Therefore,
  it can support and help automate a large number of common
  administrative tasks and host-based monitoring scenarios such as
  periodic system checks, file updates, integrity monitoring,
  patch/package management, and so on.

  This document explains how to build WebJob, configure an Apache
  Web server to handle WebJob requests, and verify client-server
  connectivity.

  WebJob is available at: http://webjob.sourceforge.net

1.1 Revision

  $Id: README.INSTALL,v 1.50 2011/01/18 04:35:25 mavrik Exp $

1.2 Table of Contents

  Section 1 .................... Overview
  Section 1.1 .................. Revision
  Section 1.2 .................. Table of Contents
  Section 1.3 .................. Caveats
  Section 1.3.1 ................ Windows NT/2K/XP Specific Caveats
  Section 2 .................... WebJob Client Installation
  Section 2.1 .................. Requirements
  Section 2.2 .................. Unpack the Source Archive
  Section 2.3 .................. Verify Installation of OpenSSL
  Section 2.4 .................. UNIX/Cygwin Build Instructions
  Section 2.4.1 ................ UNIX/Cygwin Build Gotchas
  Section 2.4.2 ................ MacOS X Universal Binaries
  Section 2.5 .................. Windows NT/2K/XP Visual Studio Build Instructions
  Section 2.5.1 ................ Windows NT/2K/XP Visual Studio Build Gotchas
  Section 2.6 .................. Windows NT/2K/XP MinGW/MSYS Build Instructions
  Section 2.6.1 ................ Windows NT/2K/XP MinGW/MSYS Build Gotchas
  Section 3 .................... WebJob Server Installation
  Section 3.1 .................. Requirements
  Section 3.2 .................. Build Instructions
  Section 3.2.1 ................ Build Gotchas
  Section 3.3 .................. Setup Instructions
  Section 3.3.1 ................ Setup Gotchas
  Section 3.4 .................. Client-Server Connectivity
  Section 3.4.1 ................ Server Configuration
  Section 3.4.2 ................ Client Configuration
  Section 3.4.3 ................ Connectivity Tests

1.3 Caveats

  The various programs and scripts referenced throughout this
  document are generally specified using relative paths.  To avoid
  potential complications, you can either make sure that your PATH
  environment variable is configured properly or provide full path
  information as needed.

1.3.1 Windows NT/2K/XP Specific Caveats

  The native NT/2K/XP build only supports execution of .exe, .com, and
  .bat files.  In particular, scripts such as .pl and .sh that
  require a file association to run will not work and are not
  supported at this time.  This limitation, however, can be overcome
  by packaging the target script as a PaD executable.

2 WebJob Client Installation

  This section describes the process for building and installing
  the WebJob client.

2.1 Requirements

  Building WebJob in UNIX or Cygwin environments requires:

    Make
      GNU or FreeBSD make is best.  Do not attempt to build with the
      native version of make under Solaris.

    GCC
      Most any version will do.  The lowest version currently used for
      test builds is 2.95.4.

    Perl
      Most any version will do, but 5.8.0 or higher is recommended.
      Note that the Perl distribution must include pod2man and
      pod2html, and if doesn't, you'll need to add them separately.

    Autoconf
      This tool is only required if you need to remake the configure
      script.  By default, the configure script is pre-built and
      included in each release.

  Building WebJob in Windows NT/2K/XP environments requires:

    MinGW+MSYS or Visual Studio
      WebJob can be built with OpenSSL support under MinGW 5.1.4 and
      MSYS 1.0.10.  Avoid lower versions of MinGW and MSYS, or be
      prepared to work through the warnings/issues.

      WebJob can be built with OpenSSL support under Visual Studio
      6.0.  Windows ACL/DACL support will be ommited from the binary
      if compiled with Visual Studio 6.0.  The recomended version
      is Visual Sutdio 8.0.

    Active Perl
      The lowest version currently used is 5.8.8 build 822.  Also, you
      must have pod2man and pod2html.  For MinGW/MSYS builds, these
      tools must be in the MSYS path.

  By default, SSL support is enabled in WebJob.  This means that
  OpenSSL must be be installed and/or available on the build system.

2.2 Unpack the Source Archive

  The source archive needs to be unpacked in a suitable location
  before the build process can commence.  All commands presented
  in this section attempt to unpack the source archive in the current
  working directory.

  On UNIX and Cygwin platforms this can be accomplished as follows:

    $ tar -zxvf <archive>

  If your version of tar doesn't support the 'z' flag, then try the
  following command pipeline:

    $ gunzip -c <archive> | tar -xvf -

  On Windows platforms WinZip may be used to unpack the archive.
  Since the source archive is actually a gzip'd tar file, WinZip
  will ask if you wish to decompress and open it -- answer yes.
  The command line syntax for WinZip is given here:

    > winzip32.exe -min -e <archive>

2.3 Verify Installation of OpenSSL

  By default, SSL support is enabled in WebJob.  This means that
  OpenSSL must be installed and/or available on the build system.
  The OpenSSL project is located at http://www.openssl.org.  Refer
  to OpenSSL's documentation for details on how to do this.

2.4 UNIX/Cygwin Build Instructions

  Typically, the build process is simply a matter of running configure
  and make.  In general, you are not required to specify any extra
  options/arguments when invoking either command.  The most basic
  build sequence is as follows:

    $ ./configure && make

  By default, SSL support is enabled.  This means that the configure
  script will attempt to locate OpenSSL on your system.  If OpenSSL
  is not found in one of several common locations, the script will
  abort.  To override this behavior and explicitly specify OpenSSL's
  location, use the --with-ssl=DIR option.  For example, if your
  copy of OpenSSL was installed in /usr/local/my_openssl, the build
  sequence would be as follows:

    $ ./configure --with-ssl=/usr/local/my_openssl && make

  To disable SSL support, use the --without-ssl option as follows:

    $ ./configure --without-ssl && make

  WebJob's default installation directory is /usr/local/webjob.
  If you want to install in a different location, set the --prefix
  configure option.  For example, if you wanted to install in the
  directory /usr/local/my_webjob, the build sequence would be as
  follows:

    $ ./configure --prefix=/usr/local/my_webjob && make

  To see a listing of all available configure options execute the
  following command:

    $ ./configure --help

  When the software has been built to your satisfaction, it can be
  tested and installed with the following commands:

    # make test
    # make install

2.4.1 UNIX/Cygwin Build Gotchas

  When experimenting with different build/configure options, be
  sure to do a "make clean-all" between builds.  This ensures that
  any intermediate files are removed.  In certain situations, build
  problems could arise if this is not done.

  Do not use the native version of make on Solaris platforms as the
  build/test process will, in all likelihood, fail.  Use GNU Make 3.80
  or higher instead (i.e., gmake).

  Executing the built-in tests (i.e., the test target) can take a
  considerable amount of time -- especially on older/slower systems.
  Also, these tests may fail if you don't have the necessary Perl
  modules installed.

  Make sure pod2man and pod2html are installed and in your PATH.  The
  documentation will not build properly without these utilities.

  Make sure strip(1) is in your PATH.  Typically, this is only a
  problem on Solaris platforms where it lives in /usr/ccs/bin.  Also,
  if /usr/ccs/bin is in your PATH, make sure it follows the path to
  GNU make -- see above for that gotcha.

2.4.2 MacOS X Universal Binaries

  To compile universal binaries on an Intel-based Mac, add the
  following configure options, and build as usual:

    --disable-dependency-tracking CFLAGS="-arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk"

2.5 Windows NT/2K/XP Visual Studio Build Instructions

  As of version 1.8.0, the preferred build process for Windows
  platforms is MinGW+MSYS.  See section 2.6 for details.

  WebJob has compiled successfully on Windows NT/2K/XP using Microsoft's
  Visual Studio 8.0.  The build process is simply a matter of setting
  up a build environment and running nmake.  Windows NT/2K/XP binaries
  are built using the following Makefile: Makefile.vs.  Note that
  there is no equivalent configure process for NT/2K/XP builds.

  When building WebJob from the command line, you may need to run
  the batch script VCVARS32.BAT to establish a build environment.
  One quick test is to check the LIB and INCLUDE variables with:

    > set LIB
    > set INCLUDE

  If neither variable is defined or you are in doubt, run
  VCVARS32.BAT.  Usually, this script is kept in Visual Studio's bin
  directory.  Once you have located the script, simply execute it.

  Because SSL support is enabled by default, you need to make sure
  that the SSL_DIR macro in Makefile.vs has an appropriate value.
  Basically, Makefile.vs was written with the assumption that your
  version of OpenSSL was installed in C:\OpenSSL.  Adjust SSL_DIR
  as necessary, if this is not the case.

  Assuming that you are required to run VCVARS32.BAT and all the
  required libraries and header files are installed in their default
  locations, the build sequence would be as follows:

    > VSVARS32.BAT (or VCVARS32.BAT for Visual Studio 6)
    > nmake -f Makefile.vs

  If you don't want SSL support, set the USE_SSL macro to 'N'.  This
  can be done from the command line as follows:

    > nmake -f Makefile.vs USE_SSL=N

  If you want to turn on embedded Perl, set the USE_PERL macro to 'Y'
  in Makefile.vs.  You will also, need to make sure that the PERL_DIR
  macro in Makefile.vs has an appropriate value.  Basically,
  Makefile.vs was written with the assumption that you version of Perl
  was installed in C:\Perl.  Adjust PERL_DIR as necessary, if this is
  not the case.  This can also be done from the command line as
  follows:

    > nmake -f Makefile.vs USE_PERL=Y

  The default value for PERL_TYPE_VERSION_BUILD is 'AP_5_8_8_822',
  which assumes that ActivePerl 5.8.8 (822) is installed on the build
  machine.  If you have a custom version of Perl installed, set the
  value of this macro to 'CUSTOM' and update PERL_COMPILER_FLAGS as
  necessary for your version of Perl.

  If you want to turn on debugging, set the BUILD_TYPE macro to
  'DEBUG'.  This can be done from the command line as follows:

    > nmake -f Makefile.vs BUILD_TYPE=DEBUG

  The various macros described in this section may be combined in any
  given build.  For example, to build a debug release without SSL
  support, invoke nmake as follows:

    > nmake -f Makefile.vs BUILD_TYPE=DEBUG USE_SSL=N

  When the software has been built to your satisfaction, it can be
  tested and installed with the following commands:

    > nmake -f Makefile.vs test
    > nmake -f Makefile.vs install

  Note: Testing requires Perl.

  WebJob's default installation directory is C:\WebJob.  If you
  want to install in a different location, adjust the INSTALL_DIR
  macro as appropriate -- either in the makefile or on the command
  line.  For example, to install WebJob in the 'Program Files'
  directory, invoke nmake as follows:

    > nmake -f Makefile.vs install INSTALL_DIR="C:\Program Files\WebJob"

2.5.1 Windows NT/2K/XP Visual Studio Build Gotchas

  You may have trouble compiling with nmake if your environment
  settings aren't correct.  When this happens, try running the batch
  file VCVARS32.BAT provided with Visual Studio.  If all else fails,
  try creating a Visual Studio project and import the source code.

  When experimenting with different build options (e.g., BUILD_TYPE,
  USE_SSL, etc.), be sure to do a "make clean-all" between builds.

    > nmake -f Makefile.vs clean-all

  This will ensure that everything is rebuilt correctly.  In certain
  situations, build problems could arise if this is not done.

  You may have trouble linking against static OpenSSL libraries.
  If that happens, check to see if the OpenSSL Makefile (ms\nt.mak)
  is using /MT -- by default it uses /MD (as of openssl-0.9.7f).

  You may have trouble with running WebJob with embedded Perl.  If
  that happens, check to see if Perl was compiled using the same
  version of Visual Studio as the one used to build WebJob.

  Executing the built-in tests (i.e., the test target) can take a
  considerable amount of time -- especially on older/slower systems.
  Also, these tests may fail if you don't have ActivePerl and the
  necessary Perl modules installed.

  Make sure pod2man and pod2html are installed and in your PATH.  The
  documentation will not build properly without these utilities.

2.6 Windows NT/2K/XP MinGW/MSYS Build Instructions

  Typically, the build process is simply a matter of running configure
  and make.  Going on the assumption that you have a static version of
  OpenSSL (0.9.8h or higher), the most basic build sequence is as
  follows:

    $ ./configure && make

  If you need to build a static version of OpenSSL, follow the
  instructions documented in this recipe:

    http://ftimes.sourceforge.net/Files/Recipes/openssl-compile-mingw.txt

  When the software has been built to your satisfaction, it can be
  tested and installed with the following commands:

    # make test
    # make install

2.6.1 Windows NT/2K/XP MinGW/MSYS Build Gotchas

  The process documented in the previous section is known to work with
  the stated versions of Active Perl, MinGW, MSYS, and OpenSSL.  Other
  combinations are not supported at this time.

  You may encounter errors building the man pages.  If that happens,
  put copies of pod2man and pod2html in MinGW's bin directory
  (typically C:\MinGW\bin).

3 WebJob Server Installation

  This section describes how to install and configure WebJob
  server-side components on a UNIX platform.  You must have Apache,
  OpenSSL, and Perl installed on the server and working correctly.

  Throughout this section, the terms username and client ID are used
  interchangeably.  By default, nph-webjob.cgi expects a client ID to
  conform to the following Perl regular expression:

    ^(?:[A-Za-z](?:(?:[0-9A-Za-z]|[_-](?=[^.]))){0,62})(?:[.][A-Za-z](?:(?:[0-9A-Za-z]|[_-](?=[^.]))){0,62}){0,127}$

3.1 Requirements

  In addition to the source distribution and standard build
  requirements documented in Section 2.1, you will need the following
  software:

    - Apache 2.0.X or 2.2.X (latest version is recommended)

    - OpenSSL 0.9.8n or higher (latest version is recommended)

    - Perl 5.8.0 or higher and the following modules:
      -- DB_File (any version)
      -- Config (any version)
      -- Cwd (any version)
      -- File::Basename (any version)
      -- File::Copy (any version)
      -- File::Path (any version)
      -- File::Temp (any version)
      -- FileHandle (2.00 or higher)
      -- MLDBM (2.01 or higher)
      -- Storable (2.09 or higher)
      -- Time::Local (any version)

  You will also need to know the following information:

    - WebJob Server Prefix (/var/webjob is recommended)

    - WWW Owner (www, apache, etc.)

    - WWW Group (www, apache, etc.)

    - Location of httpd.conf (/usr/local/apache2/conf)

    - Location of Apache certificate (/usr/local/apache2/conf/server.crt)

    - Location of bundled CAs file (/usr/local/apache2/conf/ca-bundle.crt)

    - Location of Perl interpreter (/usr/bin/perl is required)

3.2 Build Instructions

  The most basic (and recommended) build sequence is as follows:

    # ./configure --with-everything --with-server-prefix=/var/webjob
    # make
    # make test
    # make install

  If you need to customize certain variables (such as WWW owner/group,
  httpd.conf location, etc.), use './configure --help' to see which
  ones can be adjusted during the configure/build process.

3.2.1 Build Gotchas

  See Section 2.4.1.

3.3 Setup Instructions

  The following steps will setup and configure all major server-side
  components:

    # cp /var/webjob/config/server.cfg.sample /var/webjob/config/server.cfg
    # vi /var/webjob/config/server.cfg
      <finalize variables>
      :wq
    # webjob-setup-server -f /var/webjob/config/server.cfg

  Since httpd.conf was modified, you should verify its syntax as
  follows:

    # apachectl -t

  If necessary, correct syntax errors.  Otherwise, tell Apache to
  restart with:

    # apachectl -k restart

3.3.1 Setup Gotchas

  - The install location for WebJob utilities (i.e., PREFIX/bin) must
    be in your PATH.  The default PREFIX (i.e., --prefix) is
    /usr/local/webjob.

  - All Perl scripts in this distribution assume Perl (the executable)
    is installed in /usr/bin.  If this is not the case, you'll need to
    create a symbolic link from /usr/bin/perl to its real location.
    For example, suppose that Perl was installed in /usr/local/bin.
    In that case, you'd create the link as follows:

      # ln -s /usr/local/bin/perl /usr/bin/perl

  - All CGI scripts assume that WebJob server-side components are
    installed in /var/webjob.  If this is not the case, you'll need to
    modify the BaseDirectory property in various config files (e.g.,
    nph-webjob.cfg) as appropriate.

  - The default value for the AccessFileName directive is '.htaccess'.
    However, this setting is configurable, so if you are having
    htaccess issues and all other possibilities have been ruled out,
    you should check to see if this directive has been customized.

3.4 Client-Server Connectivity

  The purpose of this section is to test client-server connectivity.
  This will be done by configuring WebJob to download and run a
  script.  First, output will be directed to stdout/stderr.  If that
  works, thenoutput will be directed to the WebJob server.  Before
  tackling this section, you read through the WebJob man page.  Pay
  particular attention to examples 1 and 2.

3.4.1 Server Configuration

  On the server you'll need to create a profile for your test client.
  For UNIX platforms, use the following command:

    # webjob-create-profile -p unix -c client_1

  For Windows platforms, use the following command:

    # webjob-create-profile -p winx -c client_1

  If you installed WebJob client- and/or server-side components in
  custom locations, you may need to specify the '-H' and/or '-S'
  options like so:

    # webjob-create-profile -H /usr/local/webjob -S /var/webjob -p unix -c client_1

  Once the profile has been created, you'll need to review the config
  files that were generated, make any necessary changes, and copy them
  over to the test client.  If your server prefix is /var/webjob, you
  should have the following files:

    /var/webjob/profiles/client_1/etc/upload-a.cfg
    /var/webjob/profiles/client_1/etc/upload-a.ptr
    /var/webjob/profiles/client_1/etc/upload-b.cfg
    /var/webjob/profiles/client_1/etc/upload-b.ptr
    /var/webjob/profiles/client_1/etc/upload.cfg

    /var/webjob/profiles/client_1/workers/daily.cfg
    /var/webjob/profiles/client_1/workers/hourly.cfg

3.4.2 Client Configuration

  On the client you'll to install WebJob (see Section 2) and deploy
  the config files created in the previous section.  When done, you
  should have a tree (relative to prefix) similar to the one shown
  here:

    prefix (/usr/local/webjob)
      |
      + bin
      |   |
      |   - webjob
      |
      + dsv
      |   |
      |   - webjob-dsv.pem (conditionally required)
      |
      + etc
      |   |
      |   - custom-a.cfg (optional)
      |   - custom-b.cfg (optional)
      |   - upload-a.cfg
      |   - upload-a.ptr
      |   - upload-b.cfg
      |   - upload-b.ptr
      |   - upload.cfg
      |   - webjob-execute.cfg.sample
      |   - webjob-get-url.cfg.sample
      |
      + workers
      |   |
      |   - daily.cfg
      |   - hourly.cfg
      |
      + man
      |   |
      |   - man1
      |       |
      |       - webjob.1
      |
      + run

  Make sure that config files are only readable by the user that will
  be executing webjob (typically root).

3.4.3 Connectivity Tests

  Now, it's time to conduct connectivity tests.  Execute one of the
  commands listed below from a shell on the client.  Output will be
  directed to stdout/stderr.  Note that you may need to adjust your
  PATH if the shell can't locate the webjob binary.

    $ webjob --execute --file upload.cfg --NoUpload -- testenv

  or (for Windows)

    > webjob --execute --file upload.cfg --NoUpload -- testenv.cmd

  You should see output similar to this on your terminal screen:

    PATH=<...>
    WEBJOB_CFG_FILE=/usr/local/webjob/etc/upload.cfg
    WEBJOB_CLIENTID=client_1
    WEBJOB_HOSTNAME=somehost
    WEBJOB_PRIORITY=
    WEBJOB_QUEUETAG=
    WEBJOB_TEMP_DIRECTORY=/usr/local/webjob/run
    WEBJOB_WORK_DIRECTORY=/usr/local/webjob/run/webjob_1229490020_03493075

  If that worked, remove the '--NoUpload' option, and run the commadn
  again.  This time, WebJob will attempt to upload its output to the
  server.  If the upload succeeds, there will be four files in the
  server's incoming directory having the extensions: .env, .err, .out,
  and .rdy.

  If these files don't exist on the server, check the exit code ($?
  for UNIX environments).  Also, check Apache and WebJob log files for
  any signs of trouble.  On the client, there are several controls may
  help you troubleshoot the problem (see man page): StrictControls,
  OverwriteExecutable, UnlinkExecutable, UnlinkOutput, and
  UploadOnGetFailure.

