Labels and manual evidence

rqm show and rqm test give every requirement one word, its label. The label says where the requirement stands once the latest run and the baseline are put together. It is computed every time from stored facts, it is not stored anywhere, and it cannot be written into a requirement: a file that has a verified key on a requirement is refused like any other unknown key.

The inputs

The label is a function of four things and nothing else:

  • whether test is one of the verification methods of the requirement,
  • the status of the requirement,
  • the baseline state (see the page on the baseline file): current, stale, absent, invalid or disabled,
  • the run state: the evaluation of the latest run, if that run is still about the requirement as it is now.

A run is still about the requirement when it evaluated the present effective hash. A run that evaluated an older text counts as no run, so a green run from before an edit never makes the edited requirement look verified. The run state is then none. Otherwise it is the evaluation of the run: satisfied, incomplete, unsatisfied or unknown.

The rules

The first rule that matches decides.

# Condition Label
1 test is not a method not_applicable
1a the status is deprecated or rejected not_gated
2 the baseline is invalid unknown
3 the baseline is current and the run is none or satisfied, or the baseline is disabled and the run is satisfied verified, or verified_draft for a draft
4 the baseline is current and the run is incomplete, unsatisfied or unknown verification_conflict
5 the baseline is stale and the run is satisfied stale, and the baseline is marked pending
6 the baseline is absent and the run is satisfied pending_baseline
7 the run is unsatisfied failed
8 the run is incomplete incomplete
9 the run is unknown unknown
10 the run is none and the baseline is stale stale
11 the run is none and the baseline is absent or disabled unverified

Rules 2 to 11 cover every combination of the five baseline states and the five run states, which a test checks cell by cell. The word verified comes from rule 3 and from nothing else, which another test checks over every input.

What each label asks of you

Label Meaning
verified A record in the baseline matches the present text, and the latest run, if there is one for this text, agrees.
verified_draft The same, for a requirement that is still a draft and has no baseline record to hold.
verification_conflict The baseline says verified for this text, and the latest run does not agree. The output says both.
stale The requirement, the policy or the algorithm changed after the record was written. Promote the evidence again.
pending_baseline The run is good and nothing is recorded yet. This is the normal state before the first promotion.
failed The latest run is unsatisfied.
incomplete The latest run did not produce enough to judge.
unknown The evidence cannot be judged, or the baseline cannot be read.
unverified No run says anything about this text, and no baseline record exists.
not_gated The status takes the requirement out of the gate.
not_applicable The requirement is not verified by test.

A label never changes the exit code of rqm test by itself. stale and pending_baseline are normal states before evidence is promoted, and what fails a run is the evaluation of the run.

When the label would mislead as a bare word, a sentence under it says what is behind it. For verification_conflict it says when the baseline recorded the requirement as verified, which run that was, and what the latest run says now.

The boundary

Every output that shows a label ends with a note:

rqm shows that the evidence meets the declared policy. It does not show that the tests are adequate.

It is in the human output as a last line starting with note:, and in the JSON as the boundary key. A label says that evidence of the declared kind exists and is current. It cannot say that the tests exercise the requirement well, and the note is there so that no one reads it that way.

Manual evidence

A requirement with a method other than test (inspection, analysis and the others) is verified by something rqm cannot run. The author declares it in the requirement: the methods, and optionally an evidence text that says what was done.

rqm does not judge manual evidence. It does three things with it:

  • It always marks it as an assertion by the author.
  • It hashes it, so that an edit is noticed. The hash is of the methods, sorted, and the evidence text normalised like any prose of the canonical form (NFC, LF line ends, no trailing white space). The form is named manual-v1, and a change of the form would be a change of the algorithm.
  • It compares the hash with the one stored in the baseline record, if there is one, and warns with W-MANUAL-EVIDENCE-EDITED when they differ.

edited_since_record is null when there is no record to compare with, false when the hash is the same and true when it is not. It is a statement about the text and not about the work: a review that was done but not written down does not show, and an edit to a typo does.

Manual evidence never produces the label verified on its own. A requirement whose only method is manual is not_applicable, and one that has both test and a manual method is labelled from the run and the baseline like any other.