rqm coverage
rqm coverage
rqm coverage reports how many gated requirements fall into each of four classes. A gated requirement is one that is not a draft, has test among its verification methods, and has status approved or required. The four classes tell you what is missing to verify every gated requirement:
- unbound: requirements with no tests bound to them.
- unreported_source_test: requirements with tests bound to them, but some of those tests did not run.
- not_passing: requirements with tests that ran, but at least one failed.
- not_recorded: requirements that the baseline does not have a record for.
rqm coverage [--run <id>]
Each class can be in one of two states: evaluated (all required inputs are available) or not_evaluated (some required input is missing). A class marked not evaluated says nothing about its requirements. For example, not_passing is not evaluated when there is no run record, so the absence of the class from the output means nothing.
$ rqm coverage
coverage of 2 gated requirements (no run, baseline disabled)
unbound: 1
RQM-2
unreported_source_test: not evaluated (needs a run record)
not_passing: not evaluated (needs a run record)
not_recorded: not evaluated (needs a configured baseline)
note: rqm does not prove that a report accounts for every test declared in the sources. It does list the annotated tests that are absent from the report, as unreported_source_test.
note: A class marked not evaluated needs an input that is not available, and says nothing about its requirements.
note: rqm shows that the evidence meets the declared policy. It does not show that the tests are adequate.
--run <id> takes the run evaluation from a specific run instead of the latest one. An unknown run is E-RUN-NOT-FOUND.
How the classes work
- unbound is always evaluated; it says which requirements have no tests bound to them at all.
- unreported_source_test is evaluated when there is a run record; it lists requirements that have tests bound but some of them are absent from the test report.
- not_passing is evaluated when there is a run record; it lists requirements that have at least one test failure.
- not_recorded is evaluated when a baseline is configured; it lists requirements that the baseline has no record for.
rqm does not prove that a test report accounts for every test declared in the sources. It lists the annotated tests that are absent from the report, so you can see which ones were not run.
A requirement may appear in more than one class. For instance, if a test fails and the baseline has no record for the requirement, it appears in both not_passing and not_recorded. A requirement never appears in unbound and unreported_source_test at the same time, because unbound means no test is bound.
In continuous integration
To make the build fail when a coverage class is not empty, use --class <CLASS> to select the class and --strict to fail the gate:
rqm --strict coverage --class unbound
Exit codes:
- 0: All selected classes are empty or not evaluated.
- 1: At least one selected class is evaluated and not empty.
- 2: A selected class needs its input (run or baseline) which is not available, or the command line is invalid.
The --class option is repeatable to select multiple classes:
rqm --strict coverage --class unbound --class not_recorded
Without --class, every class is shown and the gate never fails (exit 1 is never returned).
Options
$ rqm coverage --help
Show which gated requirements are unbound, unreported, not passing or not recorded
Usage: rqm coverage [OPTIONS]
Options:
--format <FORMAT> Output format. JSON is the stable contract [default: human] [possible values: human, json]
--run <ID> Take the run evaluation from this run instead of the latest one
--class <CLASS> Show only this class. Repeat it to select several. Without it every class is shown [possible values: unbound, unreported_source_test, not_passing, not_recorded]
--strict Treat warnings as errors for the exit code. The output itself does not change
-h, --help Print help
The data of --format json is described in The JSON output.