Reading test reports
Reading test reports
A suite writes a JUnit XML report, and rqm reads it to learn which tests passed. A report is untrusted input: it may be huge, truncated, written by a crashed runner, or edited by someone who wants a test to count. This page says what rqm accepts, what it refuses and what it never believes.
The path from file to cases
- Read. The file is opened relative to the project and may not be a symbolic link or leave the project. If its size is over
max_report_bytes, it is refused before it is read. The read itself stops one byte past the limit, so a file that grows while it is read is refused too. - Decode. The bytes must be UTF-8, with at most one byte order mark at the start. UTF-16 and any declared encoding other than UTF-8 are refused. rqm does not guess an encoding.
- Parse. The text is parsed as XML by a parser that is set up to refuse the dangerous parts of XML (see below).
- Read the cases. The tree is read as JUnit. The element must be
testsuitesortestsuite. Counters such astests,failuresandskipped, and the attributesfileandline, are ignored: rqm counts thetestcaseelements it sees. - Adapt. The adapter of the suite turns the raw cases into logical cases, each with the identity of the test it belongs to, its attempts and its outcome.
Every step ends in a diagnostic that points into the report, or goes on.
What is refused
| Input | Result |
|---|---|
A DOCTYPE, so also any entity declaration, internal or external |
E-REPORT-MALFORMED |
| An encoding other than UTF-8, or bytes that are not UTF-8 | E-REPORT-MALFORMED at the first bad byte |
| Control characters, and character references to them or to surrogates | E-REPORT-MALFORMED |
A second root element, a root that is not testsuites or testsuite, an empty file |
E-REPORT-MALFORMED |
| Anything before the XML declaration, a second byte order mark | E-REPORT-MALFORMED |
| A truncated document | E-REPORT-MALFORMED |
| More than a limit below | E-REPORT-LIMIT |
External entities are never fetched and nothing is expanded, because a document type is refused before any entity could be used.
Limits
All limits are keys of [limits] in rqm.toml, and each must be at least 1.
| Key | Default | Counts |
|---|---|---|
max_report_bytes |
16 MiB | the size of the file |
max_report_depth |
64 | the nesting of elements |
max_report_elements |
500,000 | all elements |
max_report_attributes |
128 | attributes on one element |
max_report_attribute_bytes |
1 MiB | one attribute value |
max_report_text_bytes |
1 MiB | one run of text, such as the output of a test |
max_report_cases |
100,000 | testcase elements |
The parser library does not stop at exactly the limit it is given, so rqm checks every limit again on what the parser returns. A document that the parser calls too large or too complex is E-REPORT-LIMIT as well.
Diagnostics
E-REPORT-LIMIT: a limit above was passed. The message names the limit, the value and the key to change.E-REPORT-MALFORMED: the report is not a report rqm can read.E-REPORT-DUPLICATE-CASE: the same test appears twice in a report of a runner that should not repeat tests.
All three are exit code 2, because the input is wrong, and all three carry a line and column in the report.
Adapters
The adapter of a suite says which runner wrote the report. rqm never works it out from the file.
| Adapter | Retries in the report | Default accepted exit codes |
|---|---|---|
nextest |
child elements of the case (flakyFailure and rerunFailure), ordered by timestamp when all have one |
0 and 100 |
pytest |
none: a case that appears twice is a repeated case | 0 and 1 |
generic |
repeated testcase elements, in document order |
0 |
A case is flaky when its final attempt passed and an earlier one did not. A flaky case is a pass of the final attempt and is marked, never hidden.
Repeated cases
- nextest never repeats a case. A case that appears twice in a nextest report is
E-REPORT-DUPLICATE-CASE, and the case is in doubt. - pytest has no retry encoding. If a case appears more than once (a rerun plugin, or a collision of names), the outcome is the worst of them, the case is in doubt, and it is never a pass. When all outcomes are equal the doubt is
repeated. When they differ, the order of the attempts is unknown, and the doubt ismixed_unknown_order. - generic follows
generic_duplicate_casesin[join]:error(default) givesE-REPORT-DUPLICATE-CASEand a doubt, andattemptsreads the repeats as attempts in document order. The last one decides, and the case is flaky when the last passed after a failure.
What a report never grants
A report only says that a test with an identity ran and how it ended. rqm takes nothing else from it. A property named rqm, a requirement id in a test name, or any text in the output of a test grants no requirement. Which requirement a test belongs to is decided by the test’s declaration in the source, never by the report.
A case whose identity cannot be built is unmapped: it is counted, shown, and credited to nothing. A case that matches more than one declaration is ambiguous, and is credited to nothing either.