Testing Farm #
Testing Farm is Packit’s testing system. Test execution is managed by tmt tool.
Enable Testing #
In order to enable test execution simply include tests
jobs in the .packit.yaml
configuration:
jobs:
- job: tests
trigger: pull_request
targets:
- fedora-all
The test job by default requires Copr build to be built before running tests, and then it is installed into the testing environment.
If you want to run tests without a Copr build, the test job needs to include skip_build
(described below) option in the job configuration:
jobs:
- job: tests
trigger: pull_request
targets:
- fedora-all
skip_build: true
Required parameters:
- targets - Specify which “builds” you want to test.
As with copr_build job you can use
specific targets such as
fedora-34-x86_64
. Or just the distro part, likecentos-stream-8
, in which case the architecture isx86_64
.
You can also use the aliases provided by Packit to not need to change the config file when the new system version is released.
Each target is then mapped to a (tmt) distro
and to a Testing farm’s compose
when submitting a test. You can override the default (target to distro) mapping by
specifying targets
as a dictionary instead of as a list.
In the following example, the epel-8-x86_64
build will be tested on centos-8
distro (otherwise the default would be centos-stream-8
) and for
epel-7-x86_64
build the default mapping (to centos-7
distro) will be used:
targets:
epel-8-x86_64:
distros: [centos-8]
epel-7-x86_64: {}
Optional parameters:
- fmf_url - Git repository containing the metadata (FMF) tree. Use any format acceptable by the git clone command.
- fmf_ref - Branch, tag or commit specifying the desired git revision. Defaults to “master” when fmf_url is specified and fmf_ref is not.
- tmt_plan - Run plans by the given name. Can be passed as a regular expression.
- tf_post_install_script - Bash script as a string to run during the guest provisioning.
- tf_extra_params - a free-form dict that allows specifying extra parameters to Testing Farm. For a complete list of parameters, refer to Testing Farm documentation. The dict must follow the structure of the Testing Farm request. Options specified in the dict have the highest precedence, i.e. can override Packit’s defaults. Beware of indentation-sensitivity of the YAML format. You can verify that the option is processed correctly using a YAML parser. Refer to configuration examples for more information.
- skip_build - Whether to skip the build phase and only run tests (defaults to false). Enabling this will cause no Copr build to be built and installed into the testing environment, only submitting request to Testing Farm (the selected components to be installed should be part of the TMT definitions).
- env - A dictionary you can use to set any environment variable that will be available in the Testing Farm environment where the tests are run.
- identifier – Suffix added to the name of a GitHub check run. This is useful
when you have multiple
tests
jobs with different configuration. For example if you set this toe2e-tests
, then a check run for Rawhide would be namedtesting-farm:fedora-rawhide-x86_64:e2e-tests
.
There are also environment variables set by Packit:
PACKIT_FULL_REPO_NAME
PACKIT_UPSTREAM_NAME
PACKIT_UPSTREAM_URL
PACKIT_DOWNSTREAM_NAME
PACKIT_DOWNSTREAM_URL
PACKIT_PACKAGE_NAME
PACKIT_PACKAGE_NVR
PACKIT_BUILD_LOG_URL
PACKIT_SRPM_URL
PACKIT_COMMIT_SHA
PACKIT_COPR_PROJECT
, e.g.packit/packit-releases
PACKIT_COPR_RPMS
, space-separated list of RPMs that were built in Copr
And there are also pairs of variables for pull-request jobs:
PACKIT_SOURCE_SHA
andPACKIT_TARGET_SHA
PACKIT_SOURCE_BRANCH
andPACKIT_TARGET_BRANCH
PACKIT_SOURCE_URL
andPACKIT_TARGET_URL
Note that some variables do not need to be set if the value is unknown, irrelevant or not-configured.
Restart Testing #
The testing will automatically start after an update to the pull request
(and successful Copr build if skip_build
is false).
To trigger retesting manually (can come handy in case of infrastructure
issues for example), you can use the following comment in the pull request:
/packit test
Or if you want to re-trigger only failed tests, you can use the following comment in the pull request:
/packit retest-failed
Running tests with builds from another pull request #
It is also possible to run the tests with Copr builds built by Packit in another pull request (in a different repository). This can be useful when you are working on a change that spans multiple projects and needs to be tested together. These tests are possible to trigger only via a comment containing the argument specifying the pull request as:
/packit test <namespace>/<repo>#<pr_id>
The requirement is that in the specified PR, there were recent successful builds created by Packit for the targets configured in the repository with the “main” pull request. This is a new feature, so the behaviour may be adjusted in the future. Please reach out back to us for help or with your suggestions.
Creating Tests #
The easiest way to get started with defining tests is to use the tmt tool which will help you with the setup. Please follow tmt’s guide to get started.
Example test structure #
Once your project is initialized, this is how your structure can look like:
$ tmt
Found 3 tests: /tests/full, /tests/smoke and /tests_recording.
Found 4 plans: /plans/full, /plans/rpmlint, /plans/session-recording and /plans/smoke.
Found 0 stories.
$ ls -1 plans/
full.fmf
main.fmf
rpmlint.fmf
session-recording.fmf
smoke.fmf
More Examples #
Get inspiration for a quick start from a couple of real-life examples! These
samples live in .fmf
files inside tests or plans directories. You can also have a look
at tmt examples site.
Using Filters #
Use a custom filter
in the discover step in order to choose relevant tests only:
discover:
how: fmf
filter: "tier: 1"
url: https://src.fedoraproject.org/tests/selinux
Prepare Step #
The prepare
step can be used to define how test environment should be prepared before testing.
Provide one or more paths to ansible playbooks:
prepare:
how: ansible
playbook:
- setup/packages.yml
Apache Test #
Here is an example of a simple integration test for the web server httpd
and curl
utility:
execute:
script:
- dnf -y install httpd curl
- systemctl start httpd
- echo foo > /var/www/html/index.html
- curl http://localhost/ | grep foo
The plan above defines only the execute
step.
Individual shell commands are provided as a list.
Testing will fail if any of the commands returns a non-zero exit status.
Systemd Tests #
Below you can find a bit more interesting example of a systemd
test configuration:
summary:
Basic set of quick smoke tests for systemd.
discover:
how: fmf
filter: "tier: 1 & distro: rhel-8"
url: "https://github.com/systemd-rhel/tests"
prepare:
how: ansible
playbook: [setup/packages.yml]
execute:
how: tmt
This plan enables a set of Tier 1 tests from the shared systemd tests repository. The meaning of individual attributes is as follows:
- summary — an optional but useful attribute describing high-level purpose of the plan.
- discover — instructs to fetch tests from given repository and select relevant ones by provided
filter
. - prepare — specifies which ansible playbook should be applied to prepare environment for testing.
- execute — defines that the
tmt
should be used for running the tests.
FMF Tests #
Here’s a real-life example of tests enabled for the fmf package.
There are several plans defined under the plans directory.
The smoke
plan enables a super basic test checking availability of the fmf
command:
summary:
Just a basic smoke test
execute:
script: fmf --help
Plan features
is used to execute all available beakerlib tests from the fmf
repository:
summary:
Essential command line features
discover:
how: fmf
url: https://github.com/psss/fmf
execute:
how: tmt
It is also possible to select only a subset of available tests.
This is demonstrated by the docs
plan.
Use an fmf filter
like tier:1
to select tests for execution.
You can also reference a specific feature area instead:
summary:
Ensure that documentation is present
discover:
how: fmf
url: https://github.com/psss/fmf
filter: coverage:/stories/docs.*
execute:
how: tmt
See the stories directory to get some inspiration for organizing stories and requirements.
Running linters #
Running linters on your code is easy to set up using Testing Farm and tmt.
Linters are tools which you can install from the distribution, and they usually
just require a path to files which they check. Here is a plan which you can use
to run rpmlint
on your spec file.
rpmlint #
We are checking our spec files with rpmlint in our project:
summary:
Execute rpmlint on the spec file
discover:
how: shell
tests:
- name: rpmlint
test: rpmlint packit.spec
prepare:
- name: packages
how: install
package:
- rpmlint
execute:
how: tmt
rpminspect #
rpminspect
can analyze your packages
and give you information related to licensing, metadata, manpages, desktop app
metadata, file ownership & permissions and much much more.
Here’s a tmt plan you can use to have rpminspect invoked on SRPMs produced by Packit:
summary:
Check rpm files with rpminspect
discover:
how: shell
tests:
- name: rpminspect
test: rpminspect-fedora /tmp/*.src.rpm
prepare:
- name: packages
how: install
package:
- rpminspect
- rpminspect-data-fedora
- how: shell
script: cd /tmp && curl -O ${PACKIT_SRPM_URL}
execute:
how: tmt
You can run rpminspect also using the CentOS Stream configuration. This should prepare you before opening CentOS Stream dist-git MRs.
summary:
Check rpm files with rpminspect
discover:
how: shell
tests:
- name: rpminspect SRPM
test: rpminspect-centos -v -t VERIFY --profile=centos-stream-9-devel /tmp/*.src.rpm
prepare:
- name: packages
how: install
package:
- rpminspect
- rpminspect-data-centos
- how: shell
script: cd /tmp && curl -O ${PACKIT_SRPM_URL}
execute:
how: tmt
Since rpminspect is under active development, you should consider installing the latest version from this Copr project: https://copr.fedorainfracloud.org/coprs/dcantrell/rpminspect/
You can also inspect binary RPM files, we will make this easy to do as well in future.
Testing Farm API #
Packit Service communicates with Testing Farm via its API.
Issues & RFEs #
If you have found an issue or have an RFE, you can file an issue in nucleus project.