Reading test reports

A suite writes a JUnit XML report, and rqm reads it to learn which tests passed. A report is untrusted input: it may be huge, truncated, written by a crashed runner, or edited by someone who wants a test to count. This page says what rqm accepts, what it refuses and what it never believes.

The path from file to cases

  1. Read. The file is opened relative to the project and may not be a symbolic link or leave the project. If its size is over max_report_bytes, it is refused before it is read. The read itself stops one byte past the limit, so a file that grows while it is read is refused too.
  2. Decode. The bytes must be UTF-8, with at most one byte order mark at the start. UTF-16 and any declared encoding other than UTF-8 are refused. rqm does not guess an encoding.
  3. Parse. The text is parsed as XML by a parser that is set up to refuse the dangerous parts of XML (see below).
  4. Read the cases. The tree is read as JUnit. The element must be testsuites or testsuite. Counters such as tests, failures and skipped, and the attributes file and line, are ignored: rqm counts the testcase elements it sees.
  5. Adapt. The adapter of the suite turns the raw cases into logical cases, each with the identity of the test it belongs to, its attempts and its outcome.

Every step ends in a diagnostic that points into the report, or goes on.

What is refused

Input Result
A DOCTYPE, so also any entity declaration, internal or external E-REPORT-MALFORMED
An encoding other than UTF-8, or bytes that are not UTF-8 E-REPORT-MALFORMED at the first bad byte
Control characters, and character references to them or to surrogates E-REPORT-MALFORMED
A second root element, a root that is not testsuites or testsuite, an empty file E-REPORT-MALFORMED
Anything before the XML declaration, a second byte order mark E-REPORT-MALFORMED
A truncated document E-REPORT-MALFORMED
More than a limit below E-REPORT-LIMIT

External entities are never fetched and nothing is expanded, because a document type is refused before any entity could be used.

Limits

All limits are keys of [limits] in rqm.toml, and each must be at least 1.

Key Default Counts
max_report_bytes 16 MiB the size of the file
max_report_depth 64 the nesting of elements
max_report_elements 500,000 all elements
max_report_attributes 128 attributes on one element
max_report_attribute_bytes 1 MiB one attribute value
max_report_text_bytes 1 MiB one run of text, such as the output of a test
max_report_cases 100,000 testcase elements

The parser library does not stop at exactly the limit it is given, so rqm checks every limit again on what the parser returns. A document that the parser calls too large or too complex is E-REPORT-LIMIT as well.

Diagnostics

  • E-REPORT-LIMIT: a limit above was passed. The message names the limit, the value and the key to change.
  • E-REPORT-MALFORMED: the report is not a report rqm can read.
  • E-REPORT-DUPLICATE-CASE: the same test appears twice in a report of a runner that should not repeat tests.

All three are exit code 2, because the input is wrong, and all three carry a line and column in the report.

Adapters

The adapter of a suite says which runner wrote the report. rqm never works it out from the file.

Adapter Retries in the report Default accepted exit codes
nextest child elements of the case (flakyFailure and rerunFailure), ordered by timestamp when all have one 0 and 100
pytest none: a case that appears twice is a repeated case 0 and 1
generic repeated testcase elements, in document order 0

A case is flaky when its final attempt passed and an earlier one did not. A flaky case is a pass of the final attempt and is marked, never hidden.

Repeated cases

  • nextest never repeats a case. A case that appears twice in a nextest report is E-REPORT-DUPLICATE-CASE, and the case is in doubt.
  • pytest has no retry encoding. If a case appears more than once (a rerun plugin, or a collision of names), the outcome is the worst of them, the case is in doubt, and it is never a pass. When all outcomes are equal the doubt is repeated. When they differ, the order of the attempts is unknown, and the doubt is mixed_unknown_order.
  • generic follows generic_duplicate_cases in [join]: error (default) gives E-REPORT-DUPLICATE-CASE and a doubt, and attempts reads the repeats as attempts in document order. The last one decides, and the case is flaky when the last passed after a failure.

What a report never grants

A report only says that a test with an identity ran and how it ended. rqm takes nothing else from it. A property named rqm, a requirement id in a test name, or any text in the output of a test grants no requirement. Which requirement a test belongs to is decided by the test’s declaration in the source, never by the report.

A case whose identity cannot be built is unmapped: it is counted, shown, and credited to nothing. A case that matches more than one declaration is ambiguous, and is credited to nothing either.