# Importing Sources

## Importing already existing &debian; packages

Importing an already existing &debian; package into a &git; repository is as easy as:

      &gbp-import-dsc; package_0.1-1.dsc

This will create a new &git; repository named after the imported package, put the upstream sources onto the `upstream-branch` and the &debian; patch on the `debian-branch`. In case of a &debian; native package, only the `debian-branch` is being used. You can specify alternative branch names via the `--upstream-branch` and `--debian-branch` options, or via the `upstream-branch` and `debian-branch` options in the configuration file.

If you want to be able to exactly recreate the original tarball (orig.tar.gz) from &git;, you should also specify the `--pristine-tar` option. This is recommended.

If you want to import further versions, you can change into your shiny new &git; repository and just continue with the same command:

      cd package/
      &gbp-import-dsc; package_0.1-2.dsc
      &gbp-import-dsc; package_0.1-3.dsc
      &gbp-import-dsc; package_0.2-1.dsc

Or you can import all versions at once using &gbp-import-dscs;:

      &gbp-import-dscs; /path/to/history/package_*.dsc

This will create a &git; repository if necessary and import all versions sorted by version number.

You can also import all versions of a package known from the snapshot.debian.org service using the `--debsnap` option of &gbp-import-dscs;:

      &gbp-import-dscs; --debsnap package

## Importing a new upstream version

Change into your &git; repository (which can be empty), make sure it has all local modifications committed, and run either of:

      &gbp-import-orig; /path/to/package_0.2.orig.tar.gz
      &gbp-import-orig; /path/to/package_0.2.tar.bz2
      &gbp-import-orig; /path/to/package-0.2/

This puts the upstream sources onto the `upstream-branch` and tags them accordingly (the default tag format is `upstream/%(version)s`).

The result is then placed onto the `debian-branch`. The way this happens is determined by the `--merge-mode` option. The default mode `auto` replaces the upstream sources while preserving the `debian/` directory for 3.0 (quilt) packages. A &git; merge is used for all other source format versions.

You can again specify different branch names via the `--upstream-branch` and `--debian-branch` options.

If you are using `debian/watch` to keep track of how to retrieve upstream sources, you can also simply use the `--uscan` option:

      &gbp-import-orig; --uscan

You can also filter out content you don't want imported:

      &gbp-import-orig; --filter='CVS/*' /path/to/package_0.2.orig.tar.gz

The `--filter` option can be used multiple times for more complex filtering.

If you expect a merge conflict, you can delay the merge to the `debian-branch` via the `--no-merge` option and pull in the changes from the `upstream-branch` later.

If you want to be able to exactly recreate the original tarball (orig.tar.gz) from &git;, you should also specify the `--pristine-tar` option. This is recommended.

To customize the commit message used by &gbp-import-orig;, use the `--import-msg` option. This string is a &pyformat;, into which the `version` variable is interpolated. (i.e., use `%(version)s` in your message to get the imported upstream version).

### Recovering from errors

In case of an error &gbp-import-orig; will by default rollback (undo) all changes it has done to the repository so far:

    	$ &gbp; import-orig --verbose --filter='CVS/*' --filter='.bzr/*' --filter='.hg/*' --filter='.svn/*' --upstream-version=1.9 ../gif2apng-1.9.tar.gz
    	gbp:info: Importing '../gif2apng-1.9.tar.gz' to branch 'upstream' (filtering out ["'CVS/*'", "'.bzr/*'", "'.hg/*'", "'.svn/*'"])...
    	gbp:info: Source package is gif2apng
    	gbp:info: Upstream version is 1.9
    	gbp:info: Merging to 'master'
    	gbp:error: Automatic merge failed.
    	gbp:error: Error detected, Will roll back changes.
    	gbp:info: Rolling back branch upstream by resetting it to a6733c1f1e190ac0ed4774abc3466e9986a6df5e
    	gbp:info: Rolling back branch pristine-tar by resetting it to 0ee24ac614c920e30af82d602882c2ee841c88e5
    	gbp:info: Rolling back tag upstream/1.9 by deleting it
    	gbp:info: Rolling back branch master by resetting it to ce99782336e83a56e8e579b3cdadf93b0c19e1a8
    	gbp:info: Rolling back failed merge of upstream/1.9
    	gbp:error: Rolled back changes after import error.

In this case the import failed due to a merge conflict. Other reasons are running out of disk space, problems when generating the pristine-tar delta. If you don't want &gbp-import-orig; to undo changes made to the repository use the `--no-rollback`.

## Converting an existing &git; repository

If the &git; repository wasn't created with &gbp-import-dsc;, you have to tell &gbp-buildpackage; and friends where to find the upstream sources.

### Upstream sources on a branch

If the upstream sources are already on a separate branch, things are pretty simple. You can either rename that branch to the default `upstream-branch` name upstream with:

      &gitcmd; branch -m upstream theupstream-branch

or you can tell &gbp-buildpackage; the name of the branch to use as `upstream-branch`:

    cat <<EOF > .git/gbp.conf
    [DEFAULT]
    # this is the upstream-branch:
    upstream-branch=theupstream-branch
    EOF

If you then use &gbp-import-orig; to import new upstream sources, they will from now on end up on theupstream-branch and merged to the `debian-branch`.

### Upstream sources not on a branch

If you don't have an upstream branch but started your repository with only the upstream sources (not the &debian; patch), you can simply branch from that point. So use &gitkcmd; or &gitcmd;-log to locate the commit-id of that commit and create the upstream branch from there, e.g.:

        &gitcmd; branch upstream $(&gitcmd; log --format='%H' | tail -1)

The important thing here is that the `COMMIT_ID` specifies a point on the master branch that carried only the upstream sources and not the &debian; modifications. The above example assumes that this was the first commit to that repository.

There's currently no easy way to create the `upstream-branch` if you never had the upstream sources as a single commit. Using &gbp-import-orig; on such repositories might lead to unexpected merge results.

In order to fix this you can prepend the upstream sources as a single commit to your tree using &git;'s grafts. Afterwards you can simply create a branch as explained above and &gbp-import-orig; will work as expected.

Alternatively, if you are only importing source from original tarballs (for instance when converting from a Subversion repository where the mergeWithUpstream was set for svn-buildpackage), you can create an empty upstream branch with the following commands:

      git checkout --orphan upstream
      git rm -rf .
      git commit --allow-empty -m 'Initial upstream branch.'
      git checkout -f master
      # When not using 3.0 (quilt) with the default --merge-mode=auto
      git merge --allow-unrelated-histories upstream

## Starting a &debian; package from scratch

So far, we assumed you already have a &debian; package to start with, but what if you want to start a new package? First, create an empty repository:

      mkdir package-0.1
      cd package-0.1
      git init

Then, you import the upstream sources, branch off the `upstream-branch` branch and add the &debian; files (e.g. via dh_make):

      &gbp-import-orig; -u 0.1 ../package-0.1.tar.gz
      dh_make

That's it, you're done. If you want to publish your new repository, you can use &gbp-create-remote-repo;.

## When upstream uses Git

If upstream uses &git; for development (and you don't want to ignore that fact entirely), there are at least three ways to handle packaging. The first one uses &git; exclusively and creates the upstream tarballs from the upstream tag while the second one still uses upstream tarballs but links your packaging &git; history with upstreams &git; history. The third one also uses a tarballs but does not link to the upstream history.

### No upstream tarballs

If upstream doesn't build upstream tarballs, or you don't care about them, the simplest way is to clone upstream's repository and create a separate packaging branch in there. You will not need &gbp-import-orig; at all with this workflow. &gbp-buildpackage; will handle creating the upstream tarballs needed for the Debian source package.

For that to work you need to tell &gbp; what the upstream tag format looks like. Therefore you either use the `--git-upstream-tag` command line option or the `upstream-tag` configuration file variable to specify upstream's tag format.

For example a common upstream format is to put a `v` in front of the version number. In this case, the configuration option would look like:

    [DEFAULT]
    upstream-tag = v%(version)s

`version` will be replaced by &gbp; with the upstream version number as determined from `debian/changelog`. The `%()s` might be familiar from &pyformat;s. The option was placed in the [DEFAULT] section instead of the [buildpackage] section of the configuration so other tools like &gbp-dch; make use of it too.

Some upstreams use other formats though and don't separate numbers by dots but rather by underscore(`_`), hyphen(`-`) or anything else. In order to cope with that you can use version mangling of these characters via substitution. The substitution works as follows:

    [DEFAULT]
    upstream-tag = v%(version%.%_)s

This means that each occurrence of `.` will be replaced by `_` in the upstream version number. For example the upstream version `1.2.3` as determined from the `debian/changelog` will be looked up as &git; tag `v1_2_3` by &gbp-buildpackage;.

If you want the substitution to be the `%` character you have to escape it. E.g. `%(version%-%\%)s` will replace `-` with `%`, transforming `1-A.B.C` to `1%A.B.C`. Only a single replacement is supported and it can only replace a single character.

Since some of the possible mangling characters like `_` and `%` are also used to denote epochs and tilde revisions these versions can't be reconstructed when mapping from &git; tags back to &debian; versions and will therefore break other tools like &gbp-dch;. So use version mangling with care. It's better to come up with a Debian compatible tag format upstream. See &dep14; for the currently used expansion rules for Debian version numbers.

If you're using &pristine-tar;, you can make &gbp-buildpackage; commit the generated tarball back to the pristine-tar branch by using the `--git-pristine-tar-commit` option. This will make sure others building your package can exactly regenerate the tarball you created when building the &debian; package.

#### Step by step

To not make any assumptions about &gbp;'s configuration, the following steps have all options given in its long versions on the command line. You can add these to &gbp.conf; to save lots of typing.

First, we clone the upstream repository. To avoid any ambiguities between the &debian; packaging repository and the upstream repository, we name the upstream repository `upstream` instead of the default `origin`.

      git clone --no-checkout -o upstream git://git.example.com/libgbp.git
      cd libgbp
      git checkout -b debian/sid v1.0

The above makes sure we have `debian/sid` for the &debian; packaging. We didn't create any `upstream/*` branches; they're not needed for the packaging and only need to be kept up to date. After adding the &debian; packaging, we build the package. This assumes you're using &pristine-tar; and upstream uses a version number format as described above:

      gbp buildpackage --git-pristine-tar --git-pristine-tar-commit --git-upstream-tag='v%(version)s' --git-debian-branch=debian/sid

When updating to a new upstream version, we simply fetch from upstream and merge in the new tag. Afterwards, we update the changelog and build the package:

      git fetch upstream
      git merge v1.1
      gbp dch --debian-branch=debian/sid --snapshot --auto debian/
      &gbp-buildpackage; --git-ignore-new --git-pristine-tar --git-pristine-tar-commit --git-upstream-tag='v%(version)s'

Note that the above &gbp-dch; call makes sure we only pickup changes in the `debian/` directory. Since we told it to build a snapshot changelog entry and we hadn't commit the changelog yet, we need to tell &gbp-buildpackage; that the working directory is unclean via the `--git-ignore-new` option. Once everything looks good, commit the changelog and build a release version:

      gbp dch --release --auto --git-debian-branch=debian/sid
      git commit -m"Release 1.1-1" debian/changelog
      &gbp-buildpackage; --git-upstream-tag='v%(version)s' --git-debian-branch=debian/sid

If you want to share your repository with others, you can use &gbp-create-remote-repo; and &gbp-pull; as usual.

### Upstream tarballs and linked upstream history

If you want to track upstream's &git; but continue to import the upstream tarballs, e.g. to make sure the tarball uploaded to &debian; has the same checksum as upstream's, you can use the `--upstream-vcs-tag` option when importing new tarballs with &gbp-import-orig;. Assuming you have the upstream source in your repository with a tag `v0.0.1`, you can use:

      &gbp-import-orig; --upstream-vcs-tag=v0.0.1 foo_0.0.1.orig.tar.gz

to add upstream's tag as additional parent to the merge commit. See #664771 for more details.

### Upstream tarballs and separate upstream history

If you want to have upstream's &git; history available but don't want to link it to your packaging history you can simply keep it as a separate history. E.g. if you already have a &git; repository with your packaging, change into that repository and do:

     &gitcmd; remote add upstream https://upstream.example.com/upstream.git
     &gitcmd; fetch upstream

This will pull in upstream's &git; history into your repo but since your packaging commits and upstreams commits have no common parents the two histories will stay nicely separated. Of course you can browse it and cherry-pick from it but any remote repos you push to will not get upstream's history by default unless you push any of upstream's refs.

Since &git; has a single tag namespace pushing changes with `git push --tags` will push upstream's tags (and therefore it's history) too so be shure to only push dedicated tag names.

## Branch layout

By default, &gbp; uses one branch to keep the &debian; packaging called master and a branch to keep the upstream packaging called upstream.

This layout is simple to get started but falls short if one needs to maintain several versions of the package at the same time. Therefore the following the &dep14; layout is recommended:

debian/<release>

- the &debian; packaging for a release jessie, stretch, sid, jessie, jessie-backports, jessie-security or experimental.

upstream/latest

- the latest upstream sources.

dfsg/latest

- the DFSG-clean upstream sources in case the cleanup is done via a &git; merge from upstream to this branch.

In case &pristine-tar; is being used, there will be a single pristine-tar branch that keeps all binary deltas.