Reproduce CI environment locally

Reproduce CI environment locally #

This used to be a question in our FAQ and now we have a dedicated document to cover this.

SRPM builds locally # 

$ packit srpm

Packit will create a SRPM out of the current checkout. Simple and clear.

SRPM builds in Copr #

When your SRPM is being built in Copr (because srpm_build_deps is set in your packit config, or you installed Packit GitHub application after September 6, 2022), this section describes how you can reproduce the build procedure locally.

We invoke our CLI command packit prepare-sources in the Copr environment, the command may look like this for a job triggered by a pull request change:

$ packit prepare-sources --result-dir directory-to-place-sources --pr-id 150 
--merge-pr --target-branch main --job-config-index 2 https://github.com/packit/packit

As a first step, you can run this command locally on your computer (if you have installed the needed dependencies for your actions) and see whether the sources are correctly prepared.

Thankfully, Copr is very transparent how it performs builds. When you open a build log (e.g. builder-live.log.gz), you’ll see a command at the top which Copr invokes to perform a build. Let’s do that in a container.

$ podman run --privileged -ti fedora:36 bash

(the reason for running root privileged container is that mock will be used inside and needs capabilities to perform unshare)

But first we need to install the package which contains the copr-rpmbuild command:

[root@f26be2947a15 /]# dnf install -y copr-rpmbuild

Now let’s rerun locally a build for build-id 3580313:

[root@f26be2947a15 /]# /usr/bin/copr-rpmbuild --verbose --drop-resultdir --srpm --build-id 3580313

Your changes need to be pushed and a PR needs to be created for this to work. You are welcome to experiment with copr-rpmbuild to get a more efficient workflow (hint --task-file).

SRPM builds in our sandbox #

Packit by default runs all commands you defined in a sandbox which is a kubernetes pod in a new project. If you need additional packages or binaries present in the sandbox, you should migrate your SRPM builds to be done in Copr using srpm_build_deps.

You can reproduce our sandbox environment: Firstly, you should pull our production sandbox image and run commands of your choice inside the container. As an example, this is how we were debugging build problems with anaconda:

  1. Clone your upstream git repo.

  2. Launch the container and bind-mount the upstream project inside:

$ podman run -ti --rm --memory 768MB -v $PWD:/src -w /src quay.io/packit/sandcastle:prod bash
  1. Run commands of your choice:
[root@4af5dbd9c828 src]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '0' is supported by ustar format... yes
checking whether GID '0' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking whether make supports nested variables... (cached) yes
checking whether make supports the include directive... yes (GNU style)
checking for gcc... gcc
checking whether the C compiler works... yes
...

Our deployment is running in OpenShift ROSA.

Since OpenShift invokes pods using arbitrary UIDs and as you can see, the command above is invoked as root, it does not match production packit-service. So, if running a local container didn’t help you with reproducing the issue, you can try running it in openshift.

Here is a simple python code how packit-service does it:

from sandcastle import Sandcastle

# this should be the path to your local clone of the upstream project
git_repo_path: str = "fill-me"
# kubernetes namespace to use
k8s_namespace: str = "myproject"
command = ["your", "command", "of", "choice"]

# This is how your code gets copied (via rsync) into the openshift pod
m_dir = MappedDir(git_repo_path, "/sandcastle", with_interim_pvc=True)

o = Sandcastle(
    image_reference="docker.io/usercont/sandcastle:prod",
    k8s_namespace_name=k8s_namespace,
    mapped_dir=m_dir
)
o.run()
try:
    output = o.exec(command=command)
    print(output)
finally:
    o.delete_pod()

This script requires:

  • sandcastle installed
  • being logged in an openshift cluster (oc whoami to confirm)
  • rsync binary available

If none of these helped you, please reach out to us and we’ll try to help you.

Testing Farm #

When you open test results, you can see commands which Testing Farm performed to run your test. You can run those locally.

Testing Farm Log Output

The team is also planning to have a solution to fully reproduce the CI testing process locally: teemtee/tmt#1075.