Dar's Documentation - Building dar and libdar

# How to compile dar and libdar

Presentation
Building dar/libdar
Dar documentation
Libdar documentation
Libdar's internals

There are two ways of compiling dar. The easy way and the less easy way:

The easy way:
  This is when you grab the latest source package (file in the format dar-X.Y.Z.tar.gz where X.Y.Z is the dar version) and follow the requirements just below. The easy way also applies to interim releases when they exist, which are also fully supported and production grade software.
The less easy way:
  This is when you grab the source code from a GIT repositories. You will then have to follow the preliminary steps, then continue following the easy way.

## Requirements

To compile dar from a source package you need at least the following:

1. a C++ compiler supporting C++14 syntax (for gcc that means version 6.1. minimum, clang is also supported). For version older than 2.7.0 only C++11 is required (at least gcc version 4.9), while version older than dar-2.5.0 C++11 support is not required but if using gcc you need at least version 3.4.x. A standard C++ library is also required, compilation has been tested with libc++ and stdlibc++.
2. a linker like "ld" the GNU Linker
3. the make program (tested with gnu make)
4. pkg-config to help detecting and configuring proper CFLAGS/CXXFLAGS and LDFLAGS for optional libraries dar may relies on (see below)

In option you may also have installed the following tools and libraries:

- libz library for gzip compression support
- libbzip2 library for bzip2 compression support
- liblzo2 library for lzo compression support
- libxz library for xz/lzma compression support
- Zstandard library (version greater or equal to 1.3.0) for zstd compression support
- LZ4 library for lz4 compression support
- gnu Getopt support (Linux has it for all distro thanks to its glibc, this is not true for FreeBSD for example)
- libgcryt version 1.4.0 or greater for symetric strong encryption (blowfish, aes, etc.) and hash (sha1, md5) support
- gpgme library version 1.2.0 or greater for asymetric strong encryption and signature (RSA, DSA, etc.)
- doxygen for generation of source code documentation
- dot to generate graphes of class the C++ class hierarchy withing the doxygen documentation
- upx to generate dar_suite upx compressed binaries
- groff to generate html version of man pages
- ext2/ext3/ext4 file system libraries for Linux Filesystem Specific Attributes and nodump flag support
- libthreadar (version 1.3.0 or more recent, for MAC OS use version 1.4.0 or more recent) for libdar to use several threads compression/decompression, ciphering/deciphering as well as support for FTP repository feature (the later also needs libcurl in addition see below)
- librsync for delta binary
- libcurl for FTP repository access
- libssh (at least version 0.11.x) for SFTP repository access
- python3 (python3-dev) and pybind11 to access libdar from python
- libargon2 provides a security enhancement for key derivation fonction (strong encryption)
- librhash provides whirlpool hash algorithm

### Requirements for the Optional Features

Feature | Requirements
--- | ---
zlib compression | libz library headers and library
bzip2 compression | libbzip2 library headers and library
lzo compression | liblzo2 library headers and library
xz/lzma compression | libxz library headers and library
zstd compression | libzstd library headers and library
lz4 compression | liblz4 library headers and library
strong symmetric encryption | libgcrypt library headers and library
strong asymmetric encryption | libgcrypt library headers and library libgpgme library headers and library
documentation building | the doxygen and and dot executables at compilation time
upx-compressed dar binaries | the upx executable at compilation time
man page in html format | the groff executable at compilation time
save/restore Linux File-system Specific Attributes | libext2fs library headers and library
dar's --nodump option support | libext2fs library headers and library
FTP remote repositories support | libthreadar library annd headers libcurl library and headers
SFTP remote repositories support | libssh library and headers
binary delta support | librsync library headers and library
multi-thread compression/decompression | libthreadar library headers and library
multi-thread ciphering/deciphering | libthreadar library headers and library
python 3 API | pybind11 headers and library
key derivation function based on argon2 algorithm | libargon2 headers and library
whirlpool hash support | librhash headers and library

### Dependencies in distro packages

For simplicity, here follows the package names that provide the previously mentionned libdar dependencies. If you have the equivalent names for other distro, feel free to contact the dar's manager maintainer for this table to be updated.

Distro | Debian/Devuan/Ubuntu
--- | ---
pkg-config tool | pkg-config
libz library | zlib1g-dev
libbzip2 library | libbz2-dev
liblzo2 library | liblzo2-dev
libxz library | liblzma-dev
libzstd library | libzstd-dev
liblz4 library | liblz4-dev
libgcrypt library | libgcrypt-dev
libgpgme library | libgpgme-dev
doxygen binary | doxygen
dot binary | graphviz
upx binary | upx
groff binary | groff
libext2fs library | libext2fs-dev
libthreadar library | libthreadar-dev
librsync library | librsync-dev
libcurl library | libcurl-dev
libssh library | libssh-dev
pybind11 library | python3-pybind11 python3-dev
libargon2 library | libargon2-dev
whirlpool hash algorithm | librhash-dev

## Compilation Process

Once you have the minimum requirements, Dar has to be compiled from source code in the following way:

`./configure [eventually with some options] make make install-strip`

Important:

  due to a bug in the autoconf/libtool softwares used to build the configure script you must not have spaces in the name of the path where are extracted dar' sources. You can install dar binary anywhere you want, the problem does not concern dar itself but the `./configure` script used to build dar: To work properly it must not be ran from a path which has a space in it.

Important too:

  By default the configure script set optimization to `-O2`, depending on the compiler this may lead to problems in the resulting binary (or even in the compilation process), before reporting a bug try first to compile with less optimization:

  `CXXFLAGS=-O export CXXFLAGS make clean distclean ./configure [options...] make make install-strip`

The configure script may receive several options (listed here), in particular the `--prefix` option let you install dar and libdar in another place than the default `/usr/local`. For example to have dar installed under `/usr` use the following: `./configure --prefix=/usr`. You will be able to uninstall dar/libdar by calling `make uninstall` this implies keeping the source package directory around and using the same option given to `./configure` as what has been used at installation time.

If you prefer building a package without installing dar, which is espetially suitable for package maintainers, the `DESTDIR` variable may be set at installation time to install dar in another root directory. This makes the creation of dar binary packages very easy. Here is an example:

`./configure --prefix=/usr [eventually with some options] make make DESTDIR=/some/where install-strip`

As result of the previous example, dar will be installed in `/some/where/usr/{bin | lib | ...}` directories, but built as if it was installed in `/usr`, thus it will look for `/etc/darrc` and not `/some/where/etc/darrc`. You can thus build a package from files under `/some/where` by mean of a pakage management tool, then install/remove this package with the distro package management tools.

## Options for the `configure` script

Available options for the configure script
Optimization option: |
--enable-mode | --enable-mode=32 or --enable-mode=infinint if set,replace 64 bits integers used by default by 32 bits integers or "infinint" integers (limitless integers). Before release 2.6.0 the default integer used was infinint and this option was added for speed optimization at the cost of some limitations (See the limitations for more). Since release 2.6.0, the default is 64 bits integers (limitations stay the same) instead of infinint. But if you hit the 64 bits integer limitations you can still use infinint which overcome this at the cost of slower performance and increase memory requirement.
--enable-thread-stack-size="size" | set the subthreads stack size to the given integer argument (in byte). Expression combining numbers and multiplication operator are accepted. Without this option the default value is 8*1024*1024 (8 Mio). The main thread stack size is set by the system and is usually fine (some system even have it increasing when needed). But for threads created from libdar (for parallel compression and parallel ciphering) this depends on the Posix thread implementation. When using musl C standard library, by default new threads receive a 128 kio stack size, which does not allow libgcrypt to work, leading libdar to abort with the following error message: Error while opening libgcrypt key handle: gcrypt/Out of memory. A stack size of 8 Mio seems fine with glibc and musl standard C library (with the exception of the twofish algorithm with musl which still does not work when using parallel ciphering --- see -G option). If for some reason you want subthreads receiving a smaller or larger stack size, use this option with the expected stack size. Pay attention to the fact that choosing a too small stack size may lead to stack overflow, error, crash or other unexpected behavior, depending on the options passed to the compiler when building dar/libdar. This affects all threads created from libdar: in particular parallel compression, parallel ciphering and FTP (due to libcurl) client implementation.
Deactivation options: |
--disable-largefile | Whatever your system is, dar will not be able to handle file of size larger than 4GB
--disable-ea-support | Whatever your system is, dar will not be able to save or restore Extended Attributes (see the Notes paragraphs I and V)
--disable-nodump-flag | Whatever your system is, dar will not be able to take care of the nodump-flag (thanks to the --nodump option)
--disable-linux-statx | Even if your system provides the statx() system call, dar will ignore it and will not save birthtime of files as Linux FSA. At restoration time you will not be bothered by the warning telling you that birthime is not possible to be restored under Linux, if you still want to restore other Linux FSA.
--enable-dar-static | dar_static binary (statically linked version of dar) will be built. It may fail if a library is missing under its statically linkable version. That's up to you be sure that for each dynamically linked library detected by the configure script, a static version of that library exists. If this was not the case you have to options: Either have the configure script ignore this liibrary by mean of `--disable-...-linking` option Or add the static library beside the dynamic version (same directory) one.
--disable-special-alloc | dar uses a special allocation scheme by default (gather the many small allocations in big fewer ones), this improves dar's execution speed
--disable-upx | If upx is found in the PATH, binary are upx compressed at installation step. This can be disabled by this option, when upx is available and you don't want compressed binaries.
--disable-gnugetopt | on non GNU systems (Solaris, etc.) configure looks for libgnugetopt to have the long options support thanks to the gnu getopt_long() call, this can be disabled.
--disable-thread-safe | libdar may need POSIX mutex to be thread safe. If you don't want libdar relaying on POSIX mutex even if they are available, use this option. The resulting library may not be thread safe. But it will always be thread safe if you use --disable-special-alloc, and it will never be thread safe if --enable-test-memory is used.
--disable-libdl-linking | Ignore any libdl library and avoid linking with it
--disable-libz-linking | Disable linking to libz, thus -zgzip:* option (gzip compression) will not be available
--disable-libbz2-linking | Disable linking to libbz2, thus -zbzip2:* option (libbz2 compression) will not be available
--disable-liblzo2-linking | Disable linking to liblzo2, thus -zlzo:* option (lzo compression) will not be available
--disable-libxz-linking | Disable linking to liblzma5 this -zxz:* option (xz compression) will not be available
--disable-libgcrypt-linking | Disable linking with libgcrypt library. Strong encryption will not be available neither a hashing of generated slices.
--disable-gpgme-linking | Disable linking with gpgme library. Asymetric strong encryption algorithms will not be available
--disable-build-html | Do not build API documentation reference with Doxygen (when it is available)
--disable-furtive-read | Do not try to detect whether the system does support furtive read mode. This will lead furtive read mode to stay disabled in any case.
--disable-fast-dir | Disable optimization for large directories, doing so has a little positive impact on memory requirement but a huge drawback on execution time
--disable-execinfo | Disable reporting stack information on self diagnostic bugs even
--disable-threadar | Avoid linking with libthreadar even if available, libdar will not create threads
--disable-birthtime | Disable the HFS+ Filesystem Specific Attribute support
--disable-librsync-linking | Disable linking with librsync, thus delta binary will not be available
--disable-libcurl-linking | Disable linking with libcurl, thus remote repository support using ftp will not be available
--disable-libssh-linking | Disable linking with libssh, thus remote repository support using sftp will not be available
--disable-librhash-linking | Disable linking with librhash, thus whirlpool hash will not be available
--enable-limit-time-accuracy={s|us|ns} | Limit the timestamp precision of files to seconds, microseconds (lowercase U not the μ greek letter for μs) and nanoseconds respectively, by default dar uses the maximum time precision supported by the operating system.
Troubleshooting option: |
--enable-os-bits | If set, dar uses the given argument (32 or 64) to determine which integer type to use. This much match your CPU register size. By default dar uses the system <stdint.h> file to determine the correct integer type to use
Debugging options: |
--enable-examples | If set, example programs based on infinint will also be built
--enable-debug | If set, use debug compilation option, and if possible statically link binaries
--enable-pedantic | If set, transmits the -pedantic option to the compiler
--enable-profiling | Enable executable profiling
--enable-debug-memory | If set, logs all memory allocations and releases to `/tmp/dar_debug_mem_allocation.txt`. The resulting executable is expected to be very slow

## GIT

### Presentation

To manage its source code versions DAR uses GIT (it used CVS up to Q1 2012 and even the older RCS in 2002). Since September 2nd 2017, the GIT repository has been cloned to GitHub while the original repository at Sourceforge still stays updated. Both should contain the exact same code, that's up to you to choose the repository you prefer. For sanity also, starting September 3rd 2017, git tags including those for release candidates will be signed with GPG.

### Dar's repository Organization

GIT (more than CVS) eases the use of branches. In dar repository, there are thus a lot of them: the first and main one is called "master". It contains current development and most probably unstable code. There are other permanent branches that hold stable code. They are all named by "branch_A.B.x" where A and B are the numbers corresponding to a released versions family. For example, "branch_2.6.x" holds the stable code for releases 2.6.0, 2.6.1, 2.6.2 and so on. It also holds pending fixes for the next release on that branch you might be interested in.

The global organisation of the repository is thus the following:

`(HEAD of "master" branch) new feature 101 | ^ | new feature 100 | ^ | new feature 99 | +--->-- fix 901 ->- fix 902 (release 2.4.1) ->- fix 903 ->- fix 904 (release 2.4.2) ->- fix 905 (HEAD of branch_2.4.x) | new feature 98 | ^ | new feature 97 | +--->-- fix 801 ->- fix 802 (release 2.3.1) (also HEAD of branch_2.3.x as no pending fix is waiting for release) | ... | ^ | initial version`

### Usage

To get dar source code from GIT you have first to clone the repository hosted at sourceforge:

`git clone https://github.com/Edrusb/DAR.git cd DAR`

You will probably not want to use current development code so you have to change from the branch master to the branch "branch_A.B.x" of you choice:

`git checkout branch_2.6.x`

That's all. You now have the most recent stable code (for branch_2.6.x in this example). To see what changes have been brought since the last release, use the following command:

`git log`

If you plan to keep the repository you've cloned, updating the change is as easy as using (no need to clone the repository from scratch again):

`git pull`

There is also a web interface to git at github

### Having the sources ready for compilation

Please read the fiile named `USING_SOURCE_FROM_GIT` located at the root of the directory tree you retrieved through GIT, it contains up to date information about the required tools and how to generate the configuration file. Then you can proceed to source compilation as done with regular source package

## Related Softwares

- TAR home page (DAR's grant and venerable brother)
- Extended Attributes home page
- GNU Privacy Guard home page
- Cygwin home page
- Parchive home page and https://github.com/BlackIkeEagle/par2cmdline (fork from the official site maintained since decembre 2013)
- the Ultimate Packer for eXecutables home page
- Doxygen home page
- Groff home page
- Gnu Make home page
- GCC home page