Running rqm in CI

rqm reads and writes local files and opens no network connection. So the parts of the work that need a network, fetching the baseline before a run and keeping it afterwards, are the job of your CI system. This page shows the arrangement that rqm is built for, and the one that its own repository uses.

The three jobs

A project that uses a baseline needs three kinds of job:

  • On every pull request: check the requirements, and run the tests. A requirement that was added or edited shows pending_baseline or stale, and that does not fail the run, so the pull request is judged by its fresh evidence. The pull request does not write the baseline.
  • After a merge to the main branch: run the tests again and promote their evidence into the baseline, with a job that has the right to store the file. Because only that job writes, nobody can forge a verified record by editing a file in a pull request.
  • Before a release: run the release gate, rqm check --baseline, which fails when a required requirement does not have a current record.

In commands, with the baseline fetched to the path that rqm.yaml names:

rqm --strict check
rqm test --fail-on-test-failure
rqm test --fail-on-test-failure --promote --create
rqm --strict check --baseline

rqm --strict check fails on any error and on any warning. rqm test --fail-on-test-failure exits 1 when the policy says no, and also when a failing case of a mandatory suite that no requirement is bound to ran, so it is the only command a pipeline needs for the tests. rqm test --fail-on-test-failure --promote --create runs the tests, then promotes the evidence of that run, and creates the baseline when it does not exist yet. rqm --strict check --baseline reads no run record, so it says what the baseline holds and nothing else.

Where the baseline lives

The baseline is an ordinary file at the path of the baseline key of rqm.yaml. The path .rqm/baseline.yaml is a good choice: rqm init already adds .rqm/ to .gitignore, so the file cannot be committed by mistake.

rqm does not fetch it and does not upload it. A job fetches it before the run and stores it after the promotion, with whatever your system offers: an artifact store, a package registry or a separate repository. With GitLab the generic package registry works, and the job token is enough:

curl --fail --header "JOB-TOKEN: $CI_JOB_TOKEN" --output .rqm/baseline.yaml "$BASELINE_URL"
curl --fail --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file .rqm/baseline.yaml "$BASELINE_URL"

The first run has nothing to fetch, and that is fine: the job carries on, and rqm test --promote --create makes the file. If a baseline is configured and the file is missing, the commands that read it fail with E-BASELINE-MISSING, so a fetch that broke cannot pass in silence. rqm test is the one exception. It warns with W-BASELINE-MISSING and goes on, because the run is what produces the evidence for the first baseline.

One writer at a time

Two promotions at once would overwrite each other, so let your CI system run the promotion job in one place at a time, for example with a resource group. rqm helps in two ways. The run record remembers the hash of the baseline that the run was evaluated against, and rqm baseline update refuses with E-BASELINE-CHANGED when the file is not the same any more. A failed store means that the job is run again.

A release branch can keep its own baseline: give it its own location, and promote there before the gate runs. The gate then judges the evidence of the commit that is being released.

Committing the baseline instead

You can commit the baseline to your repository. Choose a path outside .rqm/, because that directory is ignored. It is not the arrangement that rqm recommends: concurrent pull requests rewrite the same file and conflict, and evidence that a contributor recorded on a machine cannot be checked by a reviewer.

What the tests need to write

The suites of rqm.yaml name the report that each one writes. For nextest that is a JUnit file that the profile of your CI run sets up in .config/nextest.toml, and rqm init prints the two lines to add. rqm decides whether a report belongs to this run by comparing its modification time with a file that it writes before it starts the suite, so a suite has to write its report on every run.

The repository of rqm

The repository of rqm uses exactly this arrangement for itself. Its requirements are all required, its tests are bound with the attribute of rqm-attr, and its pipeline runs rqm --strict check and rqm test on every push. On the main branch and on the release branch the pipeline promotes into a baseline that it keeps in the package registry of the project. The release branch ends with rqm --strict check --baseline before anything is released.