rqm test
rqm test
rqm test runs the test suites of the project, reads the reports they write, and says what the run proves about every requirement. It is the only command that starts the programs of your suites. Everything else it does, it does from the reports.
rqm test
rqm test --suite unit
rqm test --exclude e2e
Suites run one after the other, in the order of rqm.yaml. For each suite rqm starts the command, waits for it to end, and then reads the report that the suite was configured to write. See Running suites for how the process is run and ended, and Reading test reports for how a report is read.
Before anything runs
rqm test builds the whole project first, like rqm check, and it refuses to start a suite when that finds an error. A test run is expensive, and a result cannot be attributed to a requirement that did not load. Fix the errors that rqm check shows, and run again. The programs of all the suites that are going to run are looked up before the first one starts, so a command that does not exist is found at once and not after an hour of tests.
$ rqm test --suite nope
error[E-SUITE-SELECT]: The suite selection is not usable: --suite names 'nope', which is not declared in the suites section
[exit 2]
Choosing suites
Without options every suite runs. --suite names the suites to run and --exclude names suites to leave out. Both can be repeated, and the suites that run keep the order of rqm.yaml. The choice is written in the run record.
A suite that does not run gives no credit. A requirement that needs a test from a suite that was left out is incomplete, and if the requirement is required the command fails. That is on purpose: a partial run is for development, and it cannot show that the project is finished.
What a clean suite is
A suite is clean when four things are true: the process ended normally with an accepted exit code, the report was read without problems, the report was written by this run and not left over from an earlier one, and the report holds at least min_report_cases cases. Only the cases of a clean suite count as evidence. The output says which of the four failed first: process, report, stale or too_few_cases.
A suite is mandatory by default. When a mandatory suite is not clean, the command fails with the exit code of the cause. A timeout and a program that cannot be started are exit 3. An exit code that is not accepted, and a report that is missing, stale, changed while it was read, too short or unreadable, are exit 2. An optional suite that is not clean does not fail the command: its errors are replaced by one warning, W-SUITE-OPTIONAL, and the requirements that need it are incomplete.
What the run proves
Every requirement that has test in verification gets one of four results for this run:
| Result | When |
|---|---|
unsatisfied |
A case that credits it failed or ended in an error. |
unknown |
A case that credits it is an input error, such as a case that the report holds twice. |
incomplete |
A required suite is not clean, a selector in tests matched no case, there are fewer cases than min_cases, a case is ambiguous or in doubt, or only skipped cases ran. |
satisfied |
Enough cases ran, and every one of them passed. |
The first row that matches is the result. A case gets credit only from the markers in the sources and the tests selectors of the requirement, never from anything the report says about itself. A test that passed after a retry counts, with the warning W-CASE-FLAKY.
The label next to the result
Next to the result of the run, each requirement shows its label: the one word that says where the requirement stands when the run and the baseline are put together. It is computed from the run and the baseline by the fixed rules in Labels and manual evidence, and it can never be written into a requirement. A requirement that passed and has no record in the baseline is pending_baseline, and one that was edited after its record was written is stale. Both are the normal state before the evidence is promoted, so a baseline state never changes the exit code of rqm test. verification_conflict is the case where the baseline holds a record for the present text and the latest run does not agree, and then a sentence under the requirement says both.
rqm test reads the baseline, if the project has one, before it starts anything. A baseline that cannot be read or is not valid is an error (exit 2), and no suite is started, because nothing could be said about any requirement against it. A configured baseline that does not exist yet is different: the command warns with W-BASELINE-MISSING and goes on, because the run is what produces the evidence for the first baseline.
The exit code
The status of a requirement decides whether its result fails the command. approved requirements fail it when they are unsatisfied; missing evidence is reported and the command still exits 0. required requirements fail it unless they are satisfied. draft, deprecated and rejected requirements never do, and neither does a requirement without the test method. A failure of the gate is exit 1, and a worse exit code from a suite takes precedence: 2, then 3, then 1.
run 01KJ9Z6Q3K7M4N8 (loaded 4 of 4 discovered requirement files)
record .rqm/runs/01KJ9Z6Q3K7M4N8/run.yaml
suites
unit [mandatory] clean, 12 cases
requirements
RQM-1 [required] satisfied -> verified
RQM-2 [approved] incomplete -> incomplete: too_few_cases 0 of 1
satisfied 1, incomplete 1, unknown 0, unsatisfied 0
note: rqm shows that the evidence meets the declared policy. It does not show that the tests are adequate.
Failing tests that no requirement claims
By default rqm test judges only the cases that a requirement is bound to. A failing case that no requirement claims changes no verdict and does not fail the command, so a project that runs rqm test instead of its test runner loses the plain rule that a failing test fails the build. --fail-on-test-failure restores it: every failing case of a mandatory suite that was clean fails the gate and the command exits 1, whether or not a requirement is bound to it. The failing cases are listed in the failed_cases key of the data and named in the human output.
Three rules narrow what counts. Only the cases of a mandatory suite that was clean are read; an optional suite never fails the command, and a suite that is not clean is reported as before. The option looks at the report and not at the exit code of the runner, so a bound failing case still makes only its own requirement unsatisfied, and the other requirements keep their verdict. A skipped case is not a failure, and a case that passed on a later attempt is not one either. Without the option a failing test that no requirement is bound to does not fail rqm test.
The run record
Every run writes .rqm/runs/<run id>/run.yaml, once. It holds what was run and how each suite ended, the result and the reasons for every requirement, the hashes of the reports and of the inputs, and the selection. A later command reads it. The directory is not meant to be committed, and rqm init puts it in .gitignore.
rqm test does not touch the baseline. It records the path of the baseline file and the hash of its bytes, and nothing else. --promote is the exception, and the next section is about it.
Running and promoting in one step
--promote chains the promotion to the run. As soon as the run record is written, rqm promotes that run into the baseline, exactly as rqm baseline update does, with the same checks and the same atomic write. What the two commands leave behind in two steps, the one command leaves behind in one.
rqm test --promote --create
rqm test --promote --prune
--create makes the first baseline and --prune drops the records of requirements that are gone or no longer gated. Both need --promote, and the run to promote cannot be chosen: it is always the run the command just made, so --run is not an option of test. See rqm baseline for what the promotion does and what it refuses.
Two refusals come before anything runs, so no suite is started and no time is spent: a project without a baseline key has nowhere to promote into (E-BASELINE-NOT-CONFIGURED), and a configured baseline that does not exist yet is refused unless --create is given (E-BASELINE-MISSING).
The promotion is chained, not conditional on the gate. A failing run is promoted as well, and it retracts what it may; the exit code is still 1 for the failed gate. When the promotion itself is refused, for example because the baseline changed while the suites ran (E-BASELINE-CHANGED), the refusal wins over the gate: the exit code is 2 and nothing is written.
# continuous integration: run the suites, judge the run, promote it
rqm test --promote
Options
$ rqm test --help
Run the suites of the project and say what the run proves about each requirement
Usage: rqm test [OPTIONS]
Options:
--format <FORMAT> Output format. JSON is the stable contract [default: human] [possible values: human, json]
--suite <ID> Run only this suite. May be given more than once
--exclude <ID> Leave this suite out. May be given more than once
--strict Treat warnings as errors for the exit code. The output itself does not change
--fail-on-test-failure Also fail the gate (exit 1) for a failing case of a mandatory suite that no requirement is bound to. Without it only the cases of gated requirements can fail the command
--promote Then promote the evidence of this run into the baseline, as `baseline update` does
--create With --promote: create the baseline when the file does not exist yet
--prune With --promote: drop the records of requirements that are not in the project or are no longer gated
-h, --help Print help
The data of --format json is described in The JSON output.