Test identities

A test identity, written SourceTestId in the code, is the name by which rqm refers to one test declaration. It appears in the tests list of a requirement (where it is called a selector), in the index of markers that rqm builds from your source, and later in the baseline. It is a public contract: once released, an identity does not change meaning, and a change of the grammar is a migration with a note, never an edit.

The two forms

identity  := rust | other
rust      := target "::" module "::" ... "::" function
target    := crate [ "#" kind "@" package ]
other     := namespace ":" body
  • A Rust identity is the crate name, the modules from the root of the crate, and the function name, joined by ::. The root module has no name, so a test in the root of the crate alpha is alpha::plain. Every part is an ASCII identifier: letters, digits and _, not starting with a digit. A raw identifier loses its r#.
  • The crate name is the name of the Cargo target with - replaced by _. For a library it is the [lib] name, which may differ from the package name.
  • The ending #kind@package is added to the crate name only when two targets of the project have the same crate name, and then to every one of them. kind is lib, proc-macro, bin or test, and package is the name of the package, with its dashes. The target part always ends at the first ::.
  • An identity of another runner is a namespace and an opaque body, such as pytest:tests.test_db.TestTxn::test_survives_crash. The namespace is lowercase letters, digits and _, starting with a letter. The body is printable ASCII without spaces. The scan of Rust sources never produces one.

An identity never contains a parameterization. A test written with #[rstest] or #[test_case] is one declaration, and the cases that the runner builds from it are matched to it when a report is read.

Examples

Test Identity
Function plain in the root module of the library alpha (package alpha-core) alpha::plain
Function plain in db::txn of the same library alpha::db::txn::plain
A unit test of the binary alpha-core alpha_core::tests::bin_alpha_core_unit
The library beta and the binary beta of the package beta beta#lib@beta::tests::beta_lib_test and beta#bin@beta::tests::beta_bin_test
The integration test it of two packages it#test@alpha-core::alpha_it_basic and it#test@beta::beta_it_basic
A test of the integration test target custom custom::custom_target_test

The set of examples that the build checks is crates/rqm/tests/data/identity-examples.json.

How a report case finds its identity

nextest names the binary of a target by a binary id, and puts it in the classname of every case of that binary. The binary id is built from the package, not from the crate: the package name for a library, package::name for an integration test, and package::bin/name for a binary. rqm keeps a table of targets, so it does not take the crate name out of the binary id: it looks the binary id up, takes the crate name and the ending from the row, and adds the name of the case, which holds the modules and the function. If more than one target has the binary id, the case is ambiguous and is credited to nothing. rqm tests prints the table, with the binary id of every target.

pytest and generic reports do not name Rust targets, so their cases get an identity in their own namespace:

Adapter Identity of a case
pytest pytest:<classname>::<name>
generic generic:<classname>::<name>, or generic:<name> when the class is empty

The name is cleaned before it is used: it is normalized to Unicode NFC and trimmed. A parameterization is cut off at the first [ when the name ends in ] (otherwise at the first ( when it ends in )), and only if something remains before it, so test_param[a-b] and test_param[x/y] are one test, test_param, run with different parameters.

The body of an identity is printable ASCII without spaces, so every byte of the class or name outside ! to ~, and every %, is written as % and two uppercase hex digits of the UTF-8 byte. a name that ends in the letter e with an acute accent, test_unicode_ plus that letter, becomes test_unicode_%C3%A9, and adds an item becomes adds%20an%20item.

Parameterized tests. A runner builds several cases from one #[rstest] or #[test_case] declaration, and names them after it with more segments added, such as tests::t::case_1. A declaration of one of those two forms owns every case whose identity is its identity followed by :: and more. The rest is the parameter of the case. The declaration of any other form owns its identity only. If two declarations own one case, the case is ambiguous and credits nothing.

A case with an empty class in a pytest report, which is how pytest writes a collection error, has no identity and is unmapped.

What rqm sees and does not see

rqm reads the source and does not compile it. It finds functions that carry #[test], #[tokio::test], #[rstest], #[test_case] or #[quickcheck], in the files that Cargo builds as part of a target, following mod x; as the compiler does. It treats every #[cfg(...)] as true. It does not see tests that macros generate. A test whose name has a character outside ASCII cannot have an identity and is reported with E-ID-NONASCII.

Stability

The identities above are fixed at the end of the second milestone of the implementation plan, together with the namespace rule for other runners. Until then the part after a namespace is accepted but not yet restricted. After that, a change of any identity covered by the examples file fails the build, and changing the grammar needs a note in that file and a new version of it.