Testing Farm #
Enable Testing #
In order to enable test execution simply include
tests jobs in the
jobs: - job: tests trigger: pull_request metadata: targets: - fedora-all
- targets - You can use
fedora-allor release specific targets such as
centos-stream-8. The targets are mapped to Testing Farm’s composes when submitting the tests.
- fmf_url - Git repository containing the metadata (FMF) tree. Use any format acceptable by the git clone command.
- fmf_ref - Branch, tag or commit specifying the desired git revision. Defaults to “master” when fmf_url is specified and fmf_ref is not.
Restart Testing #
The testing will automatically start after an update to the pull request and successful Copr build. To trigger retesting manually (can come handy in case of infrastructure issues for example), you can use the following comment in the pull request:
Creating Tests #
More Examples #
Get inspiration for a quick start from a couple of real-life examples!
Using Filters #
Use a custom
filter in the discover step in order to choose relevant tests only:
discover: how: fmf filter: "tier: 1" repository: https://src.fedoraproject.org/tests/selinux
Prepare Step #
prepare step can be used to define how test environment should be prepared before testing.
Provide one or more paths to ansible playbooks:
prepare: how: ansible playbooks: - setup/packages.yml
Apache Test #
Here is an example of a simple integration test for the web server
/apache/smoke: execute: script: - dnf -y install httpd curl - systemctl start httpd - echo foo > /var/www/html/index.html - curl http://localhost/ | grep foo
/apache/smoke defines only the
Individual shell commands are provided as a list.
Testing will fail if any of the commands returns a non-zero exit status.
Systemd Tests #
Below you can find little bit more interesting example of a
systemd test configuration:
/systemd/smoke: summary: Basic set of quick smoke tests for systemd. discover: how: fmf filter: "tier: 1 & distro: rhel-8" repository: "https://github.com/systemd-rhel/tests" prepare: how: ansible playbooks: [setup/packages.yml] execute: how: beakerlib
This plan enables a set of Tier 1 tests from the shared systemd tests repository. The meaning of individual attributes is as follows:
- Summary — an optional but useful attribute describing high-level purpose of the plan.
- Discover — instructs to fetch tests from given repository and select relevant ones by provided
- Prepare — specifies which ansible playbook should be applied to prepare environment for testing.
- Execute — defines that the beakerlib framework should be used for running the tests.
FMF Tests #
Here’s a real-life example of tests enabled for the fmf package.
There are several plans defined under the plans directory.
smoke plan enables a super basic test checking availability of the
summary: Just a basic smoke test execute: how: shell script: fmf --help
features is used to execute all available beakerlib tests from the
summary: Essential command line features discover: how: fmf repository: https://github.com/psss/fmf execute: how: beakerlib
It is also possible to select only a subset of available tests.
This is demonstrated by the
Use an fmf
tier:1 to select tests for execution.
You can also reference a specific feature area instead:
summary: Ensure that documentation is present discover: how: fmf repository: https://github.com/psss/fmf filter: coverage:/stories/docs.* execute: how: beakerlib
See the stories directory to get some inspiration for organizing stories and requirements.
Running linters #
Running linters on your code is easy to setup using Testing Farm and tmt.
Linters are tools which you can install from the distribution and they usually
just require a path to files which they check. Here is a plan which you can use
rpmlint on your spec file:
summary: Execute rpmlint on the spec file prepare: - name: packages how: install package: - rpmlint execute: script: - rpmlint fedora/python-ogr.spec
Testing Farm API #
Packit Service communicates with Testing Farm via its API.
Issues & RFEs #
If you have found an issue or have an RFE, you can file an issue in nucleus project.