Running suites

A suite is a command that rqm starts as a child process, captures, and ends if it has to. This page says exactly what rqm does, on each platform. The library module is rqm::run, and rqm test is the command that uses it. What a run proves is on the page How a run is judged.

How a suite is started

The command of a suite is an array. The first word is the program and the rest are its arguments. There is no shell: nothing is split, expanded or interpreted, so an argument with spaces, quotes or & reaches the program as it is written.

The program is found when rqm test starts, for every selected suite, before any suite runs. A program that cannot be found is E-EXEC-NOT-FOUND and exit code 2, and nothing is started. The program that is started is always the absolute path that was found, never the bare name, and that path is what the run record shows.

  • A word with a path separator is a path, relative to the project root.
  • Any other word is searched for in PATH, in order. An empty entry and a relative entry are skipped, so the current directory is never searched by accident.
  • On Windows the search also uses PATHEXT (.COM;.EXE;.BAT;.CMD when it is not set). A file without one of those extensions is never chosen, and the current directory is never searched.
  • On Unix the file must be executable.

PATH and PATHEXT are the only variables rqm reads for this, and they are read once. Everything else is inherited by the child without change. When PATH is not set, only words with a path separator can be found.

Standard input is the null device: a read from it ends at once. The working directory is the project root.

A .cmd or .bat file may be a suite command on Windows. Arguments are escaped by the standard library, which since Rust 1.89 passes &, ", %, |, ^ and a trailing backslash as literals. An argument with a line break cannot be passed to a batch file safely and is E-EXEC-ARGUMENT at exit 2.

Output

Standard output and standard error are read while the suite runs, so a suite that writes a lot cannot stop on a full pipe. At most max_suite_output_bytes of each stream are kept for the report, from the start, and a truncated excerpt ends with ...truncated N bytes. Every byte is also written to .rqm/runs/<run_id>/<suite>.log, whatever the limit.

How a suite ends

  • It exits. The exit code and the signal are kept apart. Death by a signal is never an exit code, and on Windows an exit code from 0xC0000000 to 0xCFFFFFFF is an abnormal end and never success.
  • It runs past its time limit. On Unix the suite is a process group of its own: the group gets SIGTERM, and SIGKILL after a grace of 5 seconds. On Windows the job is terminated at once, because Windows has no request to stop that reaches every process. The suite is reported as timed out by rqm’s own record: on Windows the exit code of a process that the job ended says nothing.
  • Survivors. After a timeout, a process of the suite that is still alive 2 seconds after the kill is a survivor, and the suite is timeout_with_survivors at exit 3 (E-SUITE-SURVIVORS). On Linux a survivor is a live process of the group that is not a zombie, read from /proc. On other Unix systems rqm can only ask whether the group still exists, and a zombie that nobody has collected yet counts as alive there.
  • It leaves processes behind. If the suite exits and a process that it started is still alive 2 seconds later, whether it keeps the output open or not, rqm ends that process and reports the suite as not clean (left_processes), also E-SUITE-SURVIVORS. The same holds on every platform. If the output is still held after that and no process of the suite is alive, a process outside the tree holds it (see below), and the suite is reported the same way with no process ids.

A process that leaves its group on purpose (setsid on Unix) is outside what rqm can end or see. If it holds the output open, rqm stops waiting for the output after the settle time and says the capture is incomplete. On Windows the job does not allow breakaway, so a process that asks to leave the job fails to start.

A suite that cannot be started at all is E-EXEC-START at exit 3.

Reports that belong to one run

A report path may contain {run_id}. rqm replaces it with the id of the run and creates the directories above the report before the suite starts, so two runs never share a report. Whether a runner can be told where to write is a property of the runner, not of rqm: a runner that takes its report path on the command line can use {run_id} in its arguments, and one that writes to a fixed place cannot. For such a runner the report path is fixed and the freshness rule decides whether the report belongs to this run.

What the recorded runs of the real runners show (fixtures/):

  • pytest takes the path on the command line, --junitxml=<path>, and creates the missing directories, so {run_id} can be used in its arguments.
  • cargo-nextest writes the report to target/nextest/<profile>/<path> below the workspace, where <path> is set by the profile in a configuration file. No command-line option sets it and CARGO_TARGET_DIR does not move it, so {run_id} cannot be used and the report path of a nextest suite is fixed. The freshness rule decides whether it belongs to the run.
  • A generic runner can use {run_id} if its own command line takes the report path.

The freshness rule compares the modification time of the report with that of a marker file that rqm creates, .rqm/runs/<run_id>/<suite>.start, right before it starts the suite. It never compares with the clock, because file times and the clock do not agree closely enough. A report is fresh when it is not older than the marker. On a file system with coarse times, such as FAT or some network shares, a stale report written in the same time slot as the marker looks fresh.

rqm test reads the cases of every clean mandatory suite, whether or not a requirement is bound to them, and a failing case that no requirement is bound to fails the command only with --fail-on-test-failure.

The run record

.rqm/runs/<run_id>/run.yaml is written once and never changed. A second write for the same run is an internal error. It holds the schema, the run id, start and end time in UTC, the hash algorithm, the policy and inputs fingerprints, the parts of the inputs, the evaluated hashes, the result for each requirement, the selection, the baseline path and its bytes_hash (null when the baseline is switched off or the file does not exist), the state of git (null outside a git work tree), one entry for each suite with the program that was started, how it ended, and the state of its report, and the annotated tests that no report ran. The state of git is found by running git the same way a suite is run. How a run is judged lists the parts.