How rqm is tested
How rqm is tested
rqm is a tool that says what has been verified, so its own tests have to be the kind that could not be fooled by a green badge. This page says which kinds of test there are, and how rqm is held to its own rules.
The permanent suites
- Property tests cover hashing, the resolution of inheritance, the presentation grid and the promotion plan. The grid has twenty-five cells, five baseline states by five run states, and a test enumerates all of them.
- Golden outputs cover diagnostics and every output format.
- Golden digests pin the canonical form. Each case is one file under
crates/rqm/tests/canon/, and the digests fail when a dependency upgrade or a refactor changes any byte of the form. The fix is to undo the change or to give the algorithm a new id. - Mutation tests run on every
cargo test. They feed thousands of seeded mutations of a good baseline, and of the recorded runner reports, through the readers, and require that nothing panics and that everything refused is refused with a registered diagnostic. - Fuzz targets for the report reader and the baseline reader live in
fuzz/. They compile, and they need a nightly toolchain to run. - Guard tests read the source of rqm and fail when a rule is broken: no networking crate in
Cargo.lock, no standard networking types in the sources, no file, process, environment or clock access inverify/andpromote/, one place for each parse, a fixed number ofunsafeblocks, and no label word written as a literal outside the presentation function. - Network isolation. On Linux, a test runs every command inside a new network namespace, after checking that the namespace holds only the loopback interface, and compares the output with a run outside. It is skipped, with the reason printed, where the system refuses a namespace.
- Documentation tests. Every key of the
dataof every command must appear indocs/JSON.md, every example in the book must run and print what the page shows, and every command must have a page.
The report fixture harness
The adapters read the reports of other tools, and those tools change. The harness in xtask (cargo xtask fixtures) runs real nextest, pytest and a generic runner against small sample projects, with a test for every outcome that a runner can report: passed, failed, flaky, timed out and crashed. It records the reports and the exit codes with the versions of the tools, normalizes what varies (timestamps, durations, host names and paths) and sorts the cases.
The recorded files drive the tests of the adapters, so the normal test suite needs none of the runners installed. A scheduled pipeline on a Linux runner regenerates the files with the latest versions of the tools, and fails, keeping the difference as an artifact, when what a tool writes has changed. That is how a release of nextest or pytest that changes its JUnit output is noticed. fixtures/README.md explains how the files are made and refreshed.
How the tests are grouped
A few groups of cases are worth naming, because each one guards a promise that rqm makes:
- Verification. A current baseline plus a newer failing run. A run on a superseded hash. A satisfied run with no baseline. The labels of drafts, deprecated and rejected requirements.
- Suites. An optional suite that times out. A partial run that leaves the requirements of another suite incomplete. A runner that exits non-zero when a test fails. A selector that no credited case matches.
- Attribution. A report property that claims a requirement and is ignored. A parameterized declaration that owns
id::case_1. Two declarations that own one case. - Promotion. A baseline replaced between the run and the promotion. A promotion stopped at each of its five stages. A partial run that leaves a record of another suite alone. A plan that changes nothing and does not rewrite the file.
- Determinism. The same repository read from different directories, locales and time zones.
- Security. A link in a path, a file replaced after it was opened, a report that changes while it is read, an enormous text node, and a pipe or a device in place of a file.
rqm verifies itself
The requirements of this repository are in requirements/, and rqm checks and tests them with itself. The tests that exist are bound to them with the attribute of rqm-attr, written on the test or on the inline test module. Every requirement of the repository is required, so a requirement without a passing bound test fails the build.
In CI, every pipeline runs rqm --strict check and rqm test. On the default branch and on the release branch, the same job also runs rqm test --promote --create and stores the baseline. The release branch then runs rqm --strict check --baseline, the release gate, before anything is released. Running rqm in CI shows the arrangement.
That is evidence that the evidence meets the declared policy, and the boundary statement applies to it like to any other project: it does not show that the tests are adequate. A requirement can be satisfied by a test that does not check all of its acceptance criteria. Reading the tests against the criteria is the job of review, and rqm trace and rqm coverage are the tools for it.