Testing Farm is Packit’s testing system. It is a containerized service for running tests. Test execution is configured via Flexible Metadata Format, according to the L1 Metadata and L2 Metadata specification.
If you are interested in Testing Farm’s architecture, see the section Architecture.
Test configuration is stored directly in the project repository.
Make sure that you have
fmf package installed on your system.
In order to start, initialize the metadata tree in the root directory of your git repository.
The initialization will create
.fmf directory in your repository which you must add in your pull request otherwise Testing Farm will not find your tests.
For small projects store test config in a single
If your config grows consider using elasticity to organize your config into multiple files.
Here’s a minimal example showing how to enable a simple smoke test:
/test/build/smoke: execute: how: shell commands: - /usr/bin/binary --help
The first line defines name of the test set.
It is possible to define multiple testsets in the config to cover multiple scenarios.
Tests can be fetched from a remote repository as well.
discover step to reference the remote repository:
/test/build/smoke: discover: how: fmf repository: "https://github.com/systemd-rhel/tests" execute: how: beakerlib
filter in order to choose relevant tests only:
discover: how: fmf filter: "tier: 1 & distros: rhel-8" repository: "https://github.com/systemd-rhel/tests"
prepare step can be used to define how test environment should be prepared before testing.
Currently only ansible playbooks are supported:
prepare: how: ansible playbooks: - ci/rhel-8.yml
See the L2 Metadata specification for a detailed description of the individual test steps and the whole concept.
In order to enable test execution simply include
tests jobs in the
... jobs: - job: copr_build trigger: pull_request metadata: targets: - fedora-29-x86_64 - fedora-30-x86_64 - fedora-rawhide-x86_64 - job: tests trigger: pull_request metadata: targets: - fedora-29-x86_64 - fedora-30-x86_64 - fedora-rawhide-x86_64
That’s it! Since now you should see testing feedback in your pull requests.
The testing will be automatically started after an update to the pull request and successful copr build. To trigger retesting manually (can come handy in case of infrastructure issues for example), you can use the following comment in the pull request:
Currently supported steps and implementations:
We currently support running tests in an x86_64 virtual machine only. The system is using qemu-kvm with hardware acceleration. You have full root access in the machine. The test environment currently cannot be easily changed.
Additional machine specs:
We are working on an easy way how to develop new tests, debug and reproduce issues for failed runs. Until that is ready, your best option is to spin up a new Fedora Docker container or run the Fedora Cloud Base qcow2 manually via qemu-kvm or libvirt. See the Worker section for details how to run the tool locally.
Get inspiration for a quick start from a couple of real-life examples!
Here is an example of a simple integration test for the web server
/test/build/smoke: execute: how: shell commands: - dnf -y install httpd curl - systemctl start httpd - echo foo > /var/www/html/index.html - curl http://localhost/ | grep foo
/test/build/smoke defines only the
Individual shell commands are provided as a list.
Testing will fail if any of the commands returns a non-zero exit status.
Below you can find little bit more interesting example of a
systemd test configuration:
/test/pull-request/smoke: summary: Basic set of quick smoke tests for systemd. discover: how: fmf filter: "tier: 1 & distros: rhel-8" repository: "https://github.com/systemd-rhel/tests" prepare: how: ansible playbooks: [ci/rhel-8.yml] execute: how: beakerlib
This testset enables a set of Tier 1 tests from the shared systemd tests repository. The meaning of individual attributes is as follows:
Here’s a real-life example of tests enabled for the fmf package.
There are several testsets defined under the testsets directory.
smoke testset enables a super basic test checking availability of the
summary: Just a basic smoke test execute: how: shell commands: - fmf --help
features is used to execute all available beakerlib tests from the
summary: Essential command line features discover: how: fmf repository: https://github.com/psss/fmf execute: how: beakerlib
It is also possible to select only a subset of available tests.
This is demonstrated by the
Use an fmf
tier:1 to select tests for execution.
You can also reference a specific feature area instead:
summary: Ensure that documentation is present discover: how: fmf repository: https://github.com/psss/fmf filter: coverage:/stories/docs.* execute: how: beakerlib
See the stories directory to get some inspiration for organizing stories and requirements.
If you have found an issue or have an RFE, please use Testing Farm’s general project to file an issue.
Currently Testing Farm is deployed on Centos CI Openshift, where it runs it’s components and uses the same cluster as the testing infrastructure. See section Test Environment for more information about the test environment.
API is a REST API endpoint used for integration with other services. Packit Service uses this API to trigger tests. For each trigger request the API runs a Worker on the cluster as a separate pod.
The Console component uses it to get the logs of the Workers. Each trigger request MUST contain a unique pipeline ID, generated by the requester, which identifies the request and result and is used to access the console and testing artifacts.
The API is currently in version 0 and it is expected to change.
The source code of the API is located in the testing-farm/api project.
Is a React.js application that uses API’s console endpoint to access the Worker’s pod console.
Currently it is the main entrypoint for a user.
Packit users can access the console by clicking the
Details link in the Github PR test results.
Console provides also a link to produced test artifacts.
The source code of the Console is located in the testing-farm/console project.
Artifacts component provides access to the artifact storage to the end users. Artifacts are stored on a persistent volume storage which is shared with all the workers. Artifacts are not accessed directly, but via the Console.
The source code of the Artifacts is located in the testing-farm/artifacts project.
Worker is a container image which runs the testing workloads. Each worker is run in a separate pod.
The worker can be easily run from your localhost and if it has access to /dev/kvm, it can run basically the same workloads as CI. We will use it in the future to provide a seamless experience to the end users to reproduce the tests on your localhost.
If you want to try out worker now, you can easily run some tests from a repository with FMF tests with the command bellow.
podman run --device /dev/kvm quay.io/testing-farm/cruncher cruncher --git-url https://github.com/packit-service/hello-world --keep-instance
--keep-instance will keep the VM reserved and you will be presented in the worker logs how to connect to the machine via ssh.
The source code of the Worker is located in the testing-farm/cruncher project.