parse_junit_log is crashing with xunit from Avocado / proper xunit parsing in openQA
While I was experimenting with Avocado testing framework I have noticed a very strange behavior of openQA during parsing the results. What happens is that openQA finishes the test as incomplete and it automatically clones and restarts the test by itself. What bugs me is the restarting part, this sounds like a bug, unless this is a wanted behavior which I am not aware of.
openQA fails to parse the results, because there are differences between the junit that comes from slenkins and the xunit that comes from Acocado. For example, in the
testsuite tag, the slenkins junit format uses these extra attributes:
That has been told, if I change by hand the results produced by Avocado and add these attributes ...
- <testsuite name="avocado" tests="3" errors="0" failures="1" skipped="0" time="3.5769162178" timestamp="2016-05-04 14:46:52.803365"> + <testsuite package="avocado" hostname="localhost" id="1" disabled="0" name="avocado" tests="3" errors="0" failures="1" skipped="0" time="3.5769162178" timestamp="2016-05-04 14:46:52.803365">
... then openQA finishes the test (which is an improvement compared the previous incomplete state) but it still it's not perfect. The problem now is that openQA marks the individual testcases as failed (see here).
This is happening because of another difference between the junit from slenkins and the xunit from Avocado. This time, the difference is spotted in the
testcase tag. More specifically, slenkins uses an attribute called
status and its value is either
failure. However, this is not the same for Avocado. In Avocado there is not such
status attribute. Instead of this, they use another logic: if the test fails, there is sub-attribute called
failure. So, if the next line after a testcase is a failure, then they mark that test as failed, otherwise they mark it as passed. In order to make sure of that, I modified by hand the results of Avocado, to look like slenkins' format:
example of a test that passed:
- <testcase classname="SleepTest" name="1-sleeptest.py:SleepTest.test" time="1.00204920769"/> + <testcase classname="SleepTest" name="1-sleeptest.py:SleepTest.test" time="1.00204920769" status="success"/>
example of a test that failed:
- <testcase classname="FailTest" name="2-failtest.py:FailTest.test" time="0.00120401382446"> + <testcase classname="FailTest" name="2-failtest.py:FailTest.test" time="0.00120401382446" status="failure">
After making these changes, then openQA works as expected. It managed to parse correctly my Avocado results (see here).
If you want to test this yourself, you can copy and paste the xunit example extracted from Avocado's documentation, and write a simple test in openQA, asking it to parse this file.
assert_script_run "wget --quiet " . data_url('console/avocado.xml'); parse_junit_log("avocado.xml");
I see two solutions here:
- Either you modify the results produced by Avocado and make them look like slenkins (_IMHO I don't like this hacky way of resolving this)_.
- Enhance the
parse_junit_logfunction to understand the
#4 Updated by pgeorgiadis over 2 years ago
The openqa instance at
dede was a machine from Orthos which has been wiped since then. So, there are no data I can help you with anymore, it's been 1 year ago and I was just experimenting during hackweek. I am very sorry :/ I think what I did is just ran the basic avocado test taken from their documentation.
#5 Updated by metan over 2 years ago
This is also related to action #27726 [kernel] NVMe Over Fabrics Unit Test Framework in which we are asked to integrate nose2 based testsuite into openQA, nosetest can produce xunit as well so it would be nice if the openQA import was compatible with nose2 export too (I've started to discuss this in action #18000 but I was redirected here).
#9 Updated by EDiGiacinto about 2 years ago
PR: https://github.com/os-autoinst/os-autoinst/pull/910 - it will provide a general support to XUnit, but only xunit specification i could find were in https://github.com/lukejpreston/xunit-viewer/tree/master/data, that being said, it might require adaptations, or creation of specific parser sub-implementations depending on the framework output.
#11 Updated by EDiGiacinto about 2 years ago
- Status changed from In Progress to Resolved
Marking this as solved since now we have a more abstract interface for parsing different formats - Avocado wasn't explictly tested, but should be covered by the XUnit support we provide.
Please reopen if adaptations to the parsing of tests are needed.