This document serves as a detailed description of source-git. Please bear in mind that some things are a subject to change — the overall design is pretty solid, but details may be tinkered over time.
Authors: Stef Walter, Tomas Tomecek
Content of source-git repository is equivalent to dist-git, but uses upstream format: source files instead of tarballs, git commits instead of patches.
You can host this repository, or the specific git branch, anywhere you want. If you open a pull request, you will receive feedback on the changes:
The goal of packit is to provide automation and tooling to interact with source-git repositories so you don’t have to touch dist-git ever again. Our plan is to center development experience around upstream repositories and source-git.
Upstream repositories and source-git repositories are pretty much the same thing. Creating source-git only makes sense when the upstream does not accept downstream spec file or adding spec file to such a project doesn’t make sense.
Source git becomes the place for creative and interesting work, including aspects of packaging requiring human effort. We allow dist-git to become an auto-maintained location, used for tracking the current state of Fedora, rather than the place where any actual development happens.
Because source git is the location for creative work, we can use modern tooling, GitHub, GitLab, Pagure, pull requests, code review, continuous integration and other modern development workflows.
In many cases source git can be the upstream project git itself (mostly with projects where downstream maintainer is also the upstream maintainer). In other cases source git can be a fork of upstream git (such as with the Linux kernel).
We take cues from projects that already do this. We use the distributed nature of git repositories to overcome obstacles where certain parts of a package (patches, spec, tests) can not (due to embargoes/secrets) or will not (due the upstream project) be included in the upstream source git.
Bots are perfect candidates to perform the mundane tasks of auto-maintaining dist-git. Whenever a bot gets stuck, it can always ask maintainer for help (or the maintainer can perform the action for the bot).
One of the fundamentally useless manual activities when a maintaining a package in Fedora is moving code from one git repository format to another. a) git is distributed b) dist-git content is mostly boilerplate or regurgitated data.
Linux distributions gain an advantage from having patches incorporated upstream and not carrying them downstream.
Human effort should not be focused on repetitive automatible tasks related to churn and moving code around.
Dist-git is used as a store of state for build tools (like koji). Reinventing dist-git itself fundamentally, means reinventing a lot of tooling.
Dist-git tracks the inputs for and source state of a package build in Fedora. It is not a place for development. It is the place where integration happens.
A Dist-git branch may diverge from the stable state of a Fedora release. The stable state is represented by which builds were tagged into the stable compose, not by what is in dist-git.
Aim to do Fedora development of a package in source git. Either directly in upstream or in downstream git forks and branches of the upstream (see the kernel for longest running example of this).
Any repetitive task, whether repetitive for a single package, or repetitive across packages should be owned by bots auto-maintaining dist-git. Any creative non-automatable human task should be done in source git.
We are starting this project open source from the beginning.
In the ideal path, dist-git should be completely automatically “maintained” (already done to varying extents in the kernel, systemd, cockpit, ostree, conu, colin and other packages).
It must be trivially possible to opt in and out of auto-maintenance for a given dist-git branch.
It should continue to be possible for a human to fix up a dist-git branch, in cases where a task was done incorrectly by a bot. Bots may overwrite such fixups.
Each auto-maintained dist-git branch tracks a branch in a source git repository. The source git branch should share a common git history with the upstream project branches if maintainer desires such functionality.
Each time the HEAD of the git source branch changes, a process is started to update dist-git to reflect those changes. This process may also be triggered manually via a tool. If the dist-git is not in an expected state (last commit is not from the bot), the bot should report such divergence.
Only the most recent signed commit is a candidate for pulling into dist-git.
Source code and patches are pulled from source git branch:
Spec files are pulled from source:
%changelogin an SRPM is automatically generated from the commits in source git repository. Various techniques may be used to collapse history.
Tests are pulled from source:
After a bot makes a change to dist-git it automatically triggers the koji build.
The build in koji is validated that it works with the rest of the operating system packages in that branches compose. If it does it is then tagged into the compose.
When validation fails, feedback goes back to the upstream project. At an absolute minimum the owner of the source git change. But it must be possible to send feedback to a minimal set of Git Forges (GitHub, GitLab, etc.).
Instead of configuring the bots globally, the bot entry points (configuration) should live in the dist-git repositories (or source-git). The entry points may contain package specific code and variables that can affect the bot implementation for that repository.
Manual activities take place on source git. Humans may be involved in:
We must get credentials for the bots to perform these activities. We must implement metering in the beginning to prevent bots going wild across the entire dist-git repository.
Any change to the bots must self-validate by comparing recent bot behavior on recently changed dist-git repositories, and seeing if they have similar behavior.
In order to automate dist-git and pull from source git, an extensible configuration file would be placed in dist-git (or source-git).
Placing this config in a branch in dist-git indicates that that dist-git branch should be auto-maintained. The config may be removed to turn off auto-maintenance. There is one config per auto-maintained branch, e.g. a config in f28 dist-git branch implies the branch is auto-maintained and points to specific source git branch.
It should at a minimum support:
New upstream releases will result in new source-git branches. We can’t rebase existing branches since we would lose the provenance.
The diagram showcases how upstream releases (git tags) correspond to source git dist-git branches. New releases are automatically detected and proposed as a pull request. Once the packaging is completed, new corresponding branch is created and the new release should land in a continuously development (cont-dev) branch. Please bear in mind, that in order for a pull request to be merged, it needs to pass all the validation. Therefore in order for the 1.1.0 upstream release to land in the 1.1.0 source git branch, all the tests have to pass.
It’s up to a maintainer then to cherry-pick which changes should land in a selected downstream dist-git branch.
The actual population of dist-git and source git. Specification of the population process:
After the population process is done, the bot collects the output, performs a commit, signs it and pushes to dist-git.
These two population mechanisms (=container images) will be available to maintainers:
This is a description of the initial proposal to perform signing of commits. Our expectation is that the design will evolve over time.
The HEAD commit on the tracked branch in source git, which represents the content to land in dist-git (see above), must be signed. When new commits are pushed to source git, a bot checks signatures used to sign those commits. The signature IDs which are approved to push to dist-git need to be specified in a configuration file placed in dist-git. If the signature ID is not in the configuration file, the commit is not synced and the bot notifies owners of the source git repository.
Summary: all commits in dist-git, which are curated by a bot, are signed with bot’s key. The commit message then references the commits in source git. All the mentioned commits need to be signed so it’s possible to figure out who authored and approved the work.