Binding tests to requirements
Binding tests to requirements
A requirement that is verified by tests needs to say which tests. That is called binding a test to a requirement, and it is what lets rqm answer “is this requirement verified?” from the results of the tests and not from somebody’s word. This page describes how to bind tests, and says plainly what rqm does not check.
The tests list
A requirement lists its tests in the tests key, one selector for each test:
verification: [test]
tests:
- "sessions::tests::sign_in_with_a_valid_password"
- "sessions::tests::sign_in_with_a_wrong_password"
A selector is the identity of a test. For a Rust test, that is the name of the crate, the modules from the root of the crate, and the name of the function, joined by ::.
pub mod sessions {
// ...
#[cfg(test)]
mod tests {
#[test]
fn sign_in_with_a_valid_password() { /* ... */ }
}
}
The test above, in the library crate sessions, is sessions::tests::sign_in_with_a_valid_password. The name that cargo test prints for it is the same without the crate name. When two targets of your workspace have the same crate name, a library and a binary of one package for example, the crate name is followed by #<kind>@<package>, as in beta#lib@beta::tests::it_works, and rqm tests shows you the exact form. One test can verify several requirements, and one requirement can have several tests. A requirement does not have to list a test at all, unless it must, as the next section says.
rqm show lists the selectors of a requirement:
$ rqm show RQM-1
RQM-1 Users can sign in
uuid: <uuid>
kind: functional
status: approved
priority: must
verification: test
file: requirements/RQM-1.yaml
bytes: <hash>
effective: <hash>
presentation: unverified
baseline: disabled
text:
A user who gives a valid name and password gets a session.
acceptance:
- A valid name and password give a session.
- A wrong password gives an error and no session.
tests:
- sessions::tests::sign_in_with_a_valid_password
- sessions::tests::sign_in_with_a_wrong_password
fields:
area: auth
children:
- RQM-2 Sessions expire
linked from:
- RQM-4 Sign-in attempts are rate limited
note: rqm shows that the evidence meets the declared policy. It does not show that the tests are adequate.
What a selector looks like
A Rust selector is made of ASCII letters, digits and underscores, with :: between the parts, and no part starts with a digit. Anything else is refused when the file is read:
$ rqm check
checked 3 requirements (loaded 3 of 4 discovered requirement files)
error[E-SELECTOR-INVALID]: 'sessions::tests::the sixth attempt' is not a test selector: 'the sixth attempt' is not a module or function name: use ASCII letters, digits and '_', not starting with a digit
--> requirements/RQM-4.yaml:12:9
[exit 2]
Binding with a marker
A test can also say which requirements it implements, next to the code it tests. Add the rqm-attr crate to your dev-dependencies (cargo add --dev rqm-attr) and write the requirement ids on the test:
use rqm_attr::rqm;
#[rqm("RQM-1")]
#[test]
fn sign_in_with_a_wrong_password() { /* ... */ }
The attribute changes nothing about the test. It only checks, when your code is compiled, that its arguments are string literals, and rqm reads the rest from your source. Several ids may be written in one attribute, and the attribute may come before or after #[test]. It may also sit on an inline module, #[rqm("RQM-1")] mod tests { ... }, and then it applies to every test in the module. A marker on a test and a marker on its module are added together.
A marker is recognised by the last word of its path, so #[rqm(...)] and #[rqm_attr::rqm(...)] both count, and an attribute that you import under another name does not. Markers work on #[test], #[tokio::test], #[rstest], #[test_case] and #[quickcheck]. A marker anywhere else is E-ANNOTATION-POSITION, and that includes a module that lives in a file of its own (mod tests;): put the marker on each test there, or use selectors. A marker that names no requirement is E-ANNOTATION-UNKNOWN-REQ. rqm tests shows what each test claims:
$ rqm tests
TEST AT REQUIREMENTS
sessions::tests::sign_in_with_a_valid_password src/lib.rs:36 RQM-1
sessions::tests::sign_in_with_a_wrong_password src/lib.rs:43 RQM-1, RQM-2
2 tests in 1 targets, 2 claimed by a requirement
The two ways are added together, and a requirement is bound when either of them binds it.
What the scan can and cannot see
rqm reads your source, it does not compile it. It finds the tests that are written as functions with one of the attributes above, in the files that Cargo builds as part of a crate. It follows mod x; the way the compiler does, including mod.rs files and #[path] attributes. It cannot see a test that a macro generates, and it treats every #[cfg(...)] as true, so a test that is compiled only with some feature is listed, and a run without that feature will not report it.
When a binding is required
For most statuses a requirement may have no tests yet, and rqm reports nothing. The one exception is the status required. A requirement that is required and lists test among its verification methods must have a test bound to it, or rqm check fails with E-REQUIRED-UNBOUND. That is the point of the status: a requirement that you have said must be verified cannot be left without a way to verify it.
RQM-4 is a draft today. When it is made required, check insists on a test:
$ rqm check
checked 4 requirements (loaded 4 of 4 discovered requirement files)
error[E-REQUIRED-UNBOUND]: The requirement RQM-4 is required and verified by test, but no test is bound to it
--> requirements/RQM-4.yaml:3:5
[exit 2]
Binding a test fixes it:
$ rqm check
checked 4 requirements (loaded 4 of 4 discovered requirement files)
Only test as a verification method makes a requirement need a test. A requirement that is verified by inspection, analysis or demonstration is verified by a person, who describes in evidence what they did. RQM-3 in the example project is like that, and it can be required without any test:
$ rqm check
checked 4 requirements (loaded 4 of 4 discovered requirement files)
$ rqm list --status required
ID KIND STATUS PRIORITY TITLE
RQM-3 constraint required should Passwords are never logged
RQM-4 functional required may Sign-in attempts are rate limited
2 of 4 requirements shown
A requirement can combine methods, verification: [test, inspection]. It then needs evidence for the inspection, and, when it is required, a test for the test.
What rqm does not check
- A selector is not checked against the tests.
rqm checkchecks that a selector is well formed, and not that a test of that name exists in your code.rqm testslists the tests that exist. A selector that names a test that was renamed or deleted is caught when the tests are run: no case matches it, and the requirement isincomplete. See rqm test. - A test result is not a verification by itself.
rqm testsays what a run proves about each requirement, and the status of a requirement decides whether that fails the command. The wordverifiedis computed from the latest run and the baseline by the rules in Labels and manual evidence, and it is never written into a file by hand.