How a run is judged
How a run is judged
rqm test runs the suites, reads their reports, and gives every requirement that has test in verification a run evaluation. The evaluation says what this run proves about the requirement and nothing else. It does not look at the baseline, so it cannot depend on what an earlier run recorded. The label that puts the run and the baseline together is on the page about labels. The code that decides is in rqm::verify, which reads no file, starts no process and looks at no clock.
Which tests count for a requirement
A case of a report counts for a requirement only through two claims that are made before the run: the #[rqm("...")] markers in the sources and the tests selectors of the requirement. The credit of a case is the union of the two. A report never grants credit, whatever it says about itself.
- A case that has no identity is unmapped and credits nothing.
- A case that could belong to more than one target is ambiguous and credits nothing. A requirement that one of the candidates claims is incomplete.
- The declaration of a case is the declaration with the same identity. A test written with
#[rstest]or#[test_case]also owns every case whose identity continues its identity with::, and the rest is the parameter of the case. Other tests own their identity only. - If more than one declaration owns a case, the case is ambiguous. There is no tie-break.
- A case that no declaration owns, such as a pytest case, can still be named by a selector, which then has to match its identity exactly.
Clean suites
Only the cases of a clean suite count. A suite is clean when all four of these hold, and the output names the first that does not:
| Fact | Holds when | Otherwise |
|---|---|---|
process |
The suite ended by itself, with an accepted exit code, within its time, leaving nothing behind. | E-SUITE-EXIT, or E-SUITE-TIMEOUT or E-SUITE-SURVIVORS from the end of the suite. |
report |
The report exists, did not change while it was read, and can be read within the limits. | E-REPORT-MISSING, E-REPORT-CHANGED, E-REPORT-MALFORMED or E-REPORT-LIMIT. |
stale |
The report is not older than the start marker of the suite. | E-REPORT-STALE. |
too_few_cases |
The report holds at least min_report_cases logical cases. |
E-REPORT-TOO-SHORT. |
The accepted exit codes are accepted_exit_codes of the suite, or the default of its adapter: 0 and 100 for nextest, 0 and 1 for pytest and 0 for generic. A signal, and a Windows exit code in the abnormal range, are never accepted.
A suite is mandatory unless it says execution: optional. When a mandatory suite is not clean, the command fails with the exit code of its diagnostics. When an optional suite is not clean, its errors are replaced by the warning W-SUITE-OPTIONAL, the command is not failed by it, and the requirements that need it are incomplete.
Counting cases
min_cases counts logical cases, never attempts and never <testcase> elements. With case_granularity: parameterized every parameterization of a test is a case. With logical all the parameterizations of one declaration are one case. A case counts when it passed and nothing casts doubt on it. A skipped case ran nothing and does not count, and neither does a case in doubt. A test that passed after a failed attempt counts, with the warning W-CASE-FLAKY.
The four results
For each requirement the first row that matches is the result:
| Result | When |
|---|---|
unsatisfied |
A case that credits it, in a clean suite, failed or ended in an error. |
unknown |
A case that credits it is an input error: the report holds it twice and the policy says that is an error. |
incomplete |
A suite in require_suites is not clean or was not run, a selector matched no case, there are fewer counted cases than min_cases, a case that credits it is ambiguous, or a case is in doubt because it is repeated or its attempts disagree in an order that is not known. A requirement with skipped cases only is incomplete too. |
satisfied |
None of the above. |
A case in doubt is never a pass. A failed case and an input error beside each other give unsatisfied.
Eligibility is the first four of the six predicates that decide whether evidence may be promoted to a baseline: the requirement is gated, every suite in require_suites is clean, the evidence is sufficient, and nothing disqualifies it. The run record has the four for every requirement. A requirement is satisfied exactly when the three that are about evidence hold. The other two predicates compare a record with a later state, and are for promotion.
The status gate and the exit code
A requirement is gated when it has test in verification and a status of approved or required.
| Status | The command fails when the run evaluation is |
|---|---|
approved |
unsatisfied |
required |
anything but satisfied |
draft, deprecated and rejected requirements are never gated. A failure of the gate is exit 1. The exit code of a command is the worst of its causes in this order: 2 (invalid input, which includes a suite whose exit code is not accepted and a report that cannot be used), then 3 (an operational error: a timeout, survivors, a suite that could not be started), then 1 (the gate), then 0.
Choosing suites
--suite <id> runs only the named suites and --exclude <id> leaves suites out. A suite that does not run credits nothing, and a requirement that needs it is incomplete. The choice is written in the run record, and a name that is not declared is E-SUITE-SELECT with exit 2.
The run record
rqm test writes .rqm/runs/<run_id>/run.yaml, once, with schema 3. Besides what Running suites lists it holds:
- for each suite: the execution mode, the adapter, the report path of the run, the hash of the bytes of the report that was read, the version of the adapter, the number of cases, the number of their attempts, the number of cases that passed only after a failed attempt, and the four facts of its completeness;
- for each requirement with a run evaluation, by id: the result, the first four predicates of eligibility, the number of counted cases, the reasons, and the cases that credit it with their outcomes and the test each of them runs;
evaluated_hashes: the effective hash of each of those requirements;inputs: the parts that the inputs fingerprint is made of (the effective hash of every requirement, the hash of every source file that was read, the annotations, the selectors, and the suite configuration), andinputs_fingerprint, the hash of those parts;annotated_not_run: the annotated tests that no clean report ran.
A command that reads a record uses the one with the greatest run id, which is the latest because run ids sort by start time. rqm show takes --run <id> to use another. The reader is strict: a record with an unknown key, a missing key, another schema, a hash that is not a hash, or contents that contradict each other is E-RUN-RECORD, and nothing is taken from it.