                         ----------------------
                            Tuptime    Manual
                         ----------------------
                              version 4.0.0
                                Ricardo F.
                               28/Jul/2019



============
| Abstract |
============

Tuptime report historical and statistical real time of the system, keeping it between restarts.
Indeed, it can:

  - Count system startups
  - Register first boot time (a.k.a. installation time)
  - Count nicely and accidentally shutdowns 
  - Uptime and downtime percentage since first boot time
  - Accumulated system uptime (running and suspended), downtime and total
  - Current uptime
  - Register used kernels
  - Print formatted table or list with most of the previous values
  - Narrow reports since and/or until a given startup or timestamp
  - Reports in csv



=================
| Prerequisites |
=================

Operating systems:
   - Linux
   - BSD

Software:
   - Python 3.x
   - Python modules (most included in python core by default):
       sys, os, argparse, locale, platform, signal, logging, sqlite3, datetime

If a distro is used, all is met with:
    - In Linux:  'python' package >= 3.0
    - In FreeBSD: 'python3' and 'py36-sqlite3'



================
| Installation |
================

For Linux systems:

   + From distribution repository:

       Debian  - https://packages.debian.org/tuptime

       Ubuntu  - https://packages.ubuntu.com/tuptime

       Arch    - https://aur.archlinux.org/packages/tuptime/

   + Using .sh script:
      
       Download and execute the installation script:

           # sh <(curl -s https://raw.githubusercontent.com/rfrail3/tuptime/master/tuptime-install.sh)

   + Manual install:

       Clone repository and copy executable file:

           # install -m 755 tuptime/src/tuptime /usr/bin/tuptime

       Add tuptime user:

           # useradd --system --no-create-home --home-dir '/var/lib/tuptime' \
             --shell '/bin/false' --comment 'Tuptime execution user' tuptime

       Execute tuptime with a privilege user for create db path:

           # tuptime

       Change owner of the db path and file:

           # chown -R tuptime /var/lib/tuptime

       Copy cron file:

           # install -m 644 tuptime/src/cron.d/tuptime /etc/cron.d/tuptime

       If it use systemd, copy service file and enable it:
	
           # install -m 644 tuptime/src/systemd/tuptime.service /lib/systemd/system/tuptime.service
           # systemctl enable tuptime.service && systemctl start tuptime.service

           (Alternative) Using systemd timer instead of cron file:

               # install -m 644 tuptime/src/systemd/tuptime-cron.* /lib/systemd/system/
               # systemctl enable tuptime-cron.timer && systemctl start tuptime-cron.timer
               # rm /etc/cron.d/tuptime

       Or if it is an old RedHat system or derivate, copy the init file:

           # install -m 755 tuptime/src/init.d/redhat/tuptime /etc/init.d/tuptime
           # chkconfig --add tuptime && chkconfig tuptime on

       Or if it is and old Debian system or derivate, copy the init file:

           # install -m 755 tuptime/src/init.d/debian/tuptime /etc/init.d/tuptime
           # update-rc.d tuptime defaults
	   # /etc/init.d/tuptime start

       That's all, enjoy it.


For BSDs systems:

   + On FreeBSD and derivates using packages or ports:

           # pkg install tuptime
      
           (or)
       
           # /usr/ports/sysutils/tuptime

       Add the suggested cron entry and enable it on rc:

           # echo 'tuptime_enable="YES"' >> /etc/rc.conf

   + Manual install on FreeBSD, OpenBSD, NetBSD and derivates:

       Please, read misc/bsd-manual-install.txt


Partial support for OSX (Darwin) system:

   Tuptime is compatible with OSX, but the integration with LaunchD is not
   complete. Please, read misc/osx-notes.txt


Note about Python 2.7:

   Tuptime <= 3.5.0 can be executed with Python 2.7.
   Set the shebang line to '#!/usr/bin/env python' to get it.



===========================
| Command line parameters |
===========================

These are the command line options, no configuration file is used:

  -h, --help            show this help message and exit
  -c, --csv             csv output
  -d DATE_FORMAT, --date DATE_FORMAT
                        date format output
  --decp DECIMALS       number of decimals in percentages
  -f FILE, --filedb FILE
                        database file
  -g, --graceful        register a graceful shutdown
  -k, --kernel          print kernel information
  -l, --list            enumerate system life as list
  -n, --noup            avoid update values
  -o TYPE, --order TYPE
                        order enumerate by [<e|d|k|u|r|s>]
  -p, --power           print power states run/suspend
  -r, --reverse         reverse order
  -s, --seconds         output time in seconds and epoch
  -S SINCE, --since SINCE
                        restrict from this register number
  -t, --table           enumerate system life as table
  --tat TIMESTAMP       status at epoch timestamp
  --tsince TIMESTAMP    restrict from this epoch timestamp
  --tuntil TIMESTAMP    restrict until this epoch timestamp
  -U UNTIL, --until UNTIL
                        restrict up until this register number
  -v, --verbose         verbose output
  -V, --version         show version
  -x, --silent          update values into db without output


-h, --help
  Print a quick reference of the command line parameters.

  Examples:
         tuptime -h
         tuptime --help


-c, --csv
  Report in csv format. When this option is used, for an easier post processed,
  all lines always have the same number of values (empty or not).

  Examples:
         tuptime --csv
         tuptime --csv -t


-d <DATE_FORMAT>, --date=<DATE_FORMAT>
  Change the date format. By default it's printed based on system locales.
  Allow values are:

	%a	Weekday as locale’s abbreviated name.
	%A	Weekday as locale’s full name.
	%w	Weekday as a decimal number, where 0 is Sunday and 6 is 
		Saturday.
	%d	Day of the month as a zero-padded decimal number.
	%b	Month as locale’s abbreviated name.
	%B	Month as locale’s full name.
	%m	Month as a zero-padded decimal number.
	%y	Year without century as a zero-padded decimal number.
	%Y	Year with century as a decimal number.
	%H	Hour (24-hour clock) as a zero-padded decimal number.
	%I	Hour (12-hour clock) as a zero-padded decimal number.
	%p	Locale’s equivalent of either AM or PM.
	%M	Minute as a zero-padded decimal number.
	%S	Second as a zero-padded decimal number.
	%f	Microsecond as a decimal number, zero-padded on the left.
	%z	UTC offset in the form +HHMM or -HHMM (empty string if the 
		the object is naive).
	%Z	Time zone name (empty string if the object is naive).
	%j	Day of the year as a zero-padded decimal number.
	%U	Week number of the year (Sunday as the first day of the week) 
		as a zero padded decimal number. All days in a new year 
		preceding the first Sunday are considered to be in week 0.
	%W	Week number of the year (Monday as the first day of the week) 
		as a decimal number. All days in a new year preceding the 
		first Monday are considered to be in week 0.
	%c	Locale’s appropriate date and time representation.
	%x	Locale’s appropriate date representation.
	%X	Locale’s appropriate time representation.
	%%	A literal '%' character.


  Examples:
         tuptime -d '%X %x'  # (Default)
         tuptime -d '%H:%M:%S   %m-%d-%Y'  # British style
         tuptime -d '%H:%M:%S   %d-%b-%Y'  # Spanish style


--dec DECIMALS
  Change the decimal lenght in percentages. The number is rounded to this value.

  Examples:
         tuptime --dec 9


-f <FILE>, --filedb=<FILE>
  Define an alternative database file. Default is located in 
  '/var/lib/tuptime/tuptime.db'. It takes precedence over environmental variable
  "TUPTIME_DBF".

  Examples:
         tuptime -f /var/lib/tuptime/tuptime.db  # (Default)
         tuptime -f /tmp/test1.db
         tuptime --filedb /tmp/test2.db

  
-g, --graceful
  Register the time in db as a graceful shutdown. This option is the way that
  tuptime have for know if is a good or a bad shutdown. This is used in the 
  init.d and systemd files at the shutdown process.

  Examples:
         tuptime -g
         tuptime --graceful


-k, --kernel
  Add kernel information to the output.

  Examples:
         tuptime -k
         tuptime -lk
         tuptime --kernel
         tuptime -t --kernel


-l, --list
  Enumerate as list each startup number, startup date, uptime, shutdown date, 
  end status and offtime. Multiple order options can be combined together.

  Examples:
         tuptime -l
         tuptime --list


-n, --noup
  Avoid update values in the db with the current btime, uptime, etc. Useful 
  when reporting modified db files.

  Examples:
         tuptime -n
         tuptime --noup


-o [<d|e|k|r|s|u>], --order=[<d|e|k|r|s|u>]
  Order enumerate output from table or list by:
    <d>  downtime
    <e>  end status
    <k>  kernel
    <r>  runtime
    <s>  suspend time
    <u>  uptime
  Therefore, only works in conjunction with (-t) or (-l) options.

  Examples:
         tuptime -t -o e
         tuptime -l -o k
         tuptime -t --order u
         tuptime -l --order d


-p, --power
  Add the running and suspended accumulated time for the correspondent uptime slot.
  Note that only Python >= 3.6 can register this values.

  Examples:
         tuptime -p
         tuptime --power
         tuptime -tp
         tuptime -l --power


-r, --reverse
  Reverse the order for table or list output. Therefore, only works in conjunction with 
  any of (-t), (-l), (-e), (-o), (-u) and (-c) options.

  Examples:
         tuptime -tr
         tuptime -lur
         tuptime -teo --reverse
         tuptime -l --reverse


-s, --seconds
  Change default human readable date style and print times in seconds and
  dates in epoch.

  Examples:
         tuptime -s
         tuptime --seconds


-S <SINCE>, --since <SINCE>
  Restrict values only from this startup number. Take it from (-t) table or
  (-l) list reports. Negative values allowed.

  Examples:
          tuptime -S 34
          tuptime --since 22
          tuptime -l -S -10  # List since last 10 registers

  
-t, --table
  Enumerate as table each startup number, startup date, uptime, shutdown date, 
  end status and downtime. Multiple order options can be combined together.

  Examples:
         tuptime -t
         tuptime --table


--tat
  Report system status at specific timestamp.

  Examples:
         tuptime --tat 1555280149
         tuptime -cs --tat 1555281311


--tsince <TIMESTAMP>
  Restrict report from the given epoch timestamp. Negative values allowed.
  Options (-S) and (-U) have precedence over this one.

  Examples:
         tuptime --tsince 1454998872
         tuptime --tsince -31536000  # Since one year ago


--tuntil <TIMESTAMP>
  Restrict report until the given epoch timestamp. Negative values allowed.
  Options (-S) and (-U) have precedence over this one.

  Examples:
         tuptime --tuntil 1454999619
         tuptime --tuntil -2592000  # Until one month ago


-U <UNTIL>, --until <UNTIL>
  Restrict values only up until this startup number. Take it from (-t) table or
  (-l) list reports. Negative values allowed.

  Examples:
          tuptime -U 45
          tuptime --until 11
          tuptime -l -U -10  # List all except last 10 registers


-v, --verbose
  Print information about the internals of tuptime. It's good for debugging 
  how it gets the variables.

  Examples:
         tuptime -v
         tuptime --verbose


-V, --version:
  Print version number and exit.

  Examples:
         tuptime -V
         tuptime --version


-x, --silent
  Any output is printed. Only update values and save them into disk.

  Examples:
         tuptime -x
         tuptime --silent



=========================
| Environment variables |
=========================

List of supported environmental variables:

TUPTIME_DBF
  Set an alternative database file path. The argument -f, --filedb takes
  precedence over this.

  Example for ksh / sh / bash:
         export TUPTIME_DBF="/opt/tuptime.db"

  Example for csh / tcsh:
         setenv TUPTIME_DBF /opt/tuptime.db 

  Example for systemd unit:
         Environment="TUPTIME_DBF='/opt/tuptime.db'"



======================
| Schedule execution |
======================

It's important to have a schedule execution. If the init or systemd scripts
are installed as it needs, the program only update values at startup and 
shutdown, but if the system fails, hangs or whatever, the uptime time will
be lost. Be sure that the cron entry or systemd tuptime-cron.[timer|service]
units are installed as expected.

By default, both have a 5 minutes scheduled execution. Feel free to lower 
that value if your requirements are narrow.



==================
| Default output |
==================

System startups:	
 Total number of system startups from since to until date. Until is joined if
 is used in a narrow range.

System shutdowns:
  Total number of shutdowns done correctly or incorrectly.

System uptime:
  Percentage of uptime and time counter.

System downtime:
  Percentage of downtime and time counter.

System life:
  Time counter since first startup date until last.

Largest uptime:
Shortest uptime:
Largest downtime:
Shortest downtime:
  Time counter and date with the largest/shortest uptime register. 

Average uptime:
Average downtime: 
  Time counter with the average time.

Current uptime:
  Actual time counter and date since registered boot date.



======================================
| DB format changes between versions |
======================================

From 2.x to 3.x:
    If this error appears:
    
    ...(a few lines)...
    sqlite3.OperationalError: no such column: endst
    
    or:
    
    ...(a few lines)...
        db_rows[-1][4] = opt.endst
    IndexError: list assignment index out of range
    
    It is because the database have the format of the 2.x release. Please, execute the
    following scripts for the migration: 
    
    https://github.com/rfrail3/tuptime/blob/master/misc/cripts/db-tuptime-migrate.sh
    https://github.com/rfrail3/tuptime/blob/master/misc/cripts/db-tuptime-migrate-3.0-to-3.1.sh
 
From 3.x to 4.x:
    DB registers uptime, rntime, spdtime, downtime are now integers.
    The migrations is done automatically from Tuptime. Also, the migration script
    is located at:

    https://github.com/rfrail3/tuptime/blob/master/misc/cripts/db-tuptime-migrate-3.1-to-4.0.sh
 

==================
| Database Specs |
==================

Tuptime use a sqlite3 database located in "/var/lib/tuptime/tuptime.db" with
the following format:

tuptime (btime integer, 
         uptime integer, 
         rntime integer,
         spdtime integer,
         offbtime integer, 
         endst integer, 
         downtime integer, 
         kernel text)

btime    Startup date in epoch
uptime   Uptime seconds  
rntime   Running time seconds
spdtime  Suspend time seconds
offbtime Shutdown date in epoch
endst    Type of shutdown [1 ok | 0 bad]
downtime Downtime seconds
kernel   Name of the kernel

The number of startup sequency is extracted from "rowid", the signed integer 
key that uniquely identifies the row within its table in sqlite.

For reset all the values, simply delete the database file.

It is possible to query and modify it directly. But if a complete row is
deleted and the database is not recreate completely (vacuum), rowid keep the
real information about startup number. Tuptime will advert about it if the
verbose mode is enabled.



============================
| About sync date and time |
============================

Large jumps over the date and time in a system can cause problems, expecially
when the uptime value is low.

Take care of the harware clock RTC (if the system have), or how the operating
system initializes their clock at startup. Tuptime register the date reported by
the system at first execution after boot (usually from init system) and use it
to calculate the previous shutdown time and the current uptime value. The command
"who -b" do more less the same.

If the system starts with a incorrect date and time and some external time sync is
used, like ntpd or timesyncd, a few time after boot the startup date reported by
the system may change. A slightly drift is compensate over the uptime value, but a
very high drift can cause inconsistent registers and output, for example, Raspberry
Pi don't have any clock and initializes from 1 January 1970.

To avoid that problem the systemd unit have the time-sync.target requirement, but
for now it have a bug (https://github.com/systemd/systemd/issues/5097) and not work
as expected.

Tuptime report the drift when it is executed in verbose mode "tuptime -v" in the
line "INFO:Drift over btime". Values around +-1 are very common and can be
considered as normal.

A few recomendations / workarounds are:

If the systems is a single-board computer, like Raspberry Pi or Beaglebone, use a
RTC module and configure it properly.

Use timesyncd if is available (timedatectl set-ntp true), it can also keep the time
in systems without hardware clock.

If the system dont have any RTC and the bug over time-sync isn't fixed, the use of 
"systemd-timesyncd-wait" can reach the desired behaviour and solve it. More info
in (https://github.com/assisi/systemd-timesyncd-wait)

Use the traditional ntp client and delay the execution of tuptime until the time
is synced. Add under "[Unit]" the text "ntp.service" at the end of "After=" and
"Requires=" lines.

Force a time sync in the "tuptime.unit" file. Add under "[Service]" definition the 
line "ExecStartPre=/usr/bin/ntpdate pool.ntp.org"

In systems without systemd, "ntp-wait" can be added to the init script and the cron
line preceeding the tuptime execution. 

Try and failure is the best way to find the right solution for the environment in 
which the system is running.

Good reads:

    https://tools.ietf.org/html/rfc1589
    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=119971
    https://unix.stackexchange.com/questions/118631/how-can-i-measure-and-prevent-clock-drift
    Timekeeping in VMware Virtual Machines (Document)



========================
| Date and Epoch notes |
========================

The options --tsince and --tuntil use epoch timestamp format, as quick
reference here are some notes:

Seconds equivalences:
	1 year    31536000 seconds
	1 month    2592000 seconds
	1 week      604800 seconds
	1 day	     86400 seconds
	1 hour        3600 seconds

Convert human date to epoch:
	date --date="JAN-18-2018" +%s
	date --date="20-JAN-18" +%s
	date --date="2018-01-20 16:21:42" +%s
	date --date="2018-01-20 16:21:42 GMT+2" +%s
	date --date="3 week ago" +%s
	date --date="1 year ago" +%s
	date --date="now" +%s

Convert epoch to human date:
	date --date="@1516748400"

More info about date input formats:
	man date
	"https://www.gnu.org/software/coreutils/manual/html_node/
	Date-input-formats.html#Date-input-formats"



===========
| Caveats |
===========

Tuptime can be executed with 'root' user, but it's recommendable the use of an
unprivileged user, like 'tuptime' as the installation section reflects for
avoid security problems. In any case, if the database file is not writable by
the user that execute tuptime, the changes are not saved. This situation is
covered for print values, but not for update database, for that reason, a 
user who can write in the database file must to execute tuptime for update values
at startup and shutdown at least.

Kernel name is registered each time that tuptime update values. If ksplice is
used to update the kernel, last kernel will have the whole time assignment
since the last system start.

Note the filesystem cache on which the files resides for a while before being 
flushing into disk. If things go wrong, like a kernel panic, maybe the last 
updates of the db already written into the file can be lost.

The uptime reference is linked with the wall clock time elapsed since boot, not
the running efective time. Running (efective time) and hibernate power states
are childs of the uptime and report accumulated values inside their time range,
which counts since the startup until the shutdown.

A basic sample web wrapper is available under misc/web/. It is made in php and
can be useful as a starting template to any other particular use case.

If the system have configured a time zone with daylight saving time (DST),
to avoid problems when the time change, the real time clock (RTC) should have
set to universal time coordinated (UTC), not local time. Check it with 
'timedatectl' command.  Reports matching DST changes are completely right.
https://tldp.org/HOWTO/Clock-2.html

The name of the tool is based on the contraction of the words Total Uptime.
