rqm coverage

rqm coverage reports how many gated requirements fall into each of four classes. A gated requirement is one that is not a draft, has test among its verification methods, and has status approved or required. The four classes tell you what is missing to verify every gated requirement:

  1. unbound: requirements with no tests bound to them.
  2. unreported_source_test: requirements with tests bound to them, but some of those tests did not run.
  3. not_passing: requirements with tests that ran, but at least one failed.
  4. not_recorded: requirements that the baseline does not have a record for.
rqm coverage [--run <id>]

Each class can be in one of two states: evaluated (all required inputs are available) or not_evaluated (some required input is missing). A class marked not evaluated says nothing about its requirements. For example, not_passing is not evaluated when there is no run record, so the absence of the class from the output means nothing.

$ rqm coverage
coverage of 2 gated requirements (no run, baseline disabled)
unbound: 1
  RQM-2
unreported_source_test: not evaluated (needs a run record)
not_passing: not evaluated (needs a run record)
not_recorded: not evaluated (needs a configured baseline)
note: rqm does not prove that a report accounts for every test declared in the sources. It does list the annotated tests that are absent from the report, as unreported_source_test.
note: A class marked not evaluated needs an input that is not available, and says nothing about its requirements.
note: rqm shows that the evidence meets the declared policy. It does not show that the tests are adequate.

--run <id> takes the run evaluation from a specific run instead of the latest one. An unknown run is E-RUN-NOT-FOUND.

How the classes work

  • unbound is always evaluated; it says which requirements have no tests bound to them at all.
  • unreported_source_test is evaluated when there is a run record; it lists requirements that have tests bound but some of them are absent from the test report.
  • not_passing is evaluated when there is a run record; it lists requirements that have at least one test failure.
  • not_recorded is evaluated when a baseline is configured; it lists requirements that the baseline has no record for.

rqm does not prove that a test report accounts for every test declared in the sources. It lists the annotated tests that are absent from the report, so you can see which ones were not run.

A requirement may appear in more than one class. For instance, if a test fails and the baseline has no record for the requirement, it appears in both not_passing and not_recorded. A requirement never appears in unbound and unreported_source_test at the same time, because unbound means no test is bound.

In continuous integration

To make the build fail when a coverage class is not empty, use --class <CLASS> to select the class and --strict to fail the gate:

rqm --strict coverage --class unbound

Exit codes:

  • 0: All selected classes are empty or not evaluated.
  • 1: At least one selected class is evaluated and not empty.
  • 2: A selected class needs its input (run or baseline) which is not available, or the command line is invalid.

The --class option is repeatable to select multiple classes:

rqm --strict coverage --class unbound --class not_recorded

Without --class, every class is shown and the gate never fails (exit 1 is never returned).

Options

$ rqm coverage --help
Show which gated requirements are unbound, unreported, not passing or not recorded

Usage: rqm coverage [OPTIONS]

Options:
      --format <FORMAT>  Output format. JSON is the stable contract [default: human] [possible values: human, json]
      --run <ID>         Take the run evaluation from this run instead of the latest one
      --class <CLASS>    Show only this class. Repeat it to select several. Without it every class is shown [possible values: unbound, unreported_source_test, not_passing, not_recorded]
      --strict           Treat warnings as errors for the exit code. The output itself does not change
  -h, --help             Print help

The data of --format json is described in The JSON output.