action #16076

parse_junit_log is crashing with xunit from Avocado / proper xunit parsing in openQA

Added by pgeorgiadis about 3 years ago. Updated about 2 years ago.

Status:ResolvedStart date:18/01/2017
Priority:HighDue date:
Assignee:EDiGiacinto% Done:

0%

Category:Feature requests
Target version:Done
Difficulty:
Duration:

Description

While I was experimenting with Avocado testing framework I have noticed a very strange behavior of openQA during parsing the results. What happens is that openQA finishes the test as incomplete and it automatically clones and restarts the test by itself. What bugs me is the restarting part, this sounds like a bug, unless this is a wanted behavior which I am not aware of.

openQA fails to parse the results, because there are differences between the junit that comes from slenkins and the xunit that comes from Acocado. For example, in the testsuite tag, the slenkins junit format uses these extra attributes: package, hostname, id, disabled.

That has been told, if I change by hand the results produced by Avocado and add these attributes ...

- <testsuite name="avocado" tests="3" errors="0" failures="1" skipped="0" time="3.5769162178" timestamp="2016-05-04 14:46:52.803365">
+ <testsuite package="avocado" hostname="localhost" id="1" disabled="0" name="avocado" tests="3" errors="0" failures="1" skipped="0" time="3.5769162178" timestamp="2016-05-04 14:46:52.803365">

... then openQA finishes the test (which is an improvement compared the previous incomplete state) but it still it's not perfect. The problem now is that openQA marks the individual testcases as failed (see here).

This is happening because of another difference between the junit from slenkins and the xunit from Avocado. This time, the difference is spotted in the testcase tag. More specifically, slenkins uses an attribute called status and its value is either success or failure. However, this is not the same for Avocado. In Avocado there is not such status attribute. Instead of this, they use another logic: if the test fails, there is sub-attribute called failure. So, if the next line after a testcase is a failure, then they mark that test as failed, otherwise they mark it as passed. In order to make sure of that, I modified by hand the results of Avocado, to look like slenkins' format:

example of a test that passed:

- <testcase classname="SleepTest" name="1-sleeptest.py:SleepTest.test" time="1.00204920769"/>
+ <testcase classname="SleepTest" name="1-sleeptest.py:SleepTest.test" time="1.00204920769" status="success"/>

example of a test that failed:

- <testcase classname="FailTest" name="2-failtest.py:FailTest.test" time="0.00120401382446">
+ <testcase classname="FailTest" name="2-failtest.py:FailTest.test" time="0.00120401382446" status="failure">

After making these changes, then openQA works as expected. It managed to parse correctly my Avocado results (see here).

If you want to test this yourself, you can copy and paste the xunit example extracted from Avocado's documentation, and write a simple test in openQA, asking it to parse this file.

For example:

assert_script_run "wget --quiet " . data_url('console/avocado.xml');
parse_junit_log("avocado.xml");

I see two solutions here:

  1. Either you modify the results produced by Avocado and make them look like slenkins (_IMHO I don't like this hacky way of resolving this)_.
  2. Enhance the parse_junit_log function to understand the xunit from Avocado.

More information about xunit/json in Avocado, can be found here and there.

History

#1 Updated by RBrownSUSE about 3 years ago

  • Subject changed from parse_junit_log is crashing with xunit from Avocado to [tools]parse_junit_log is crashing with xunit from Avocado

#2 Updated by coolo over 2 years ago

  • Subject changed from [tools]parse_junit_log is crashing with xunit from Avocado to parse_junit_log is crashing with xunit from Avocado
  • Assignee deleted (szarate)
  • Priority changed from Normal to High
  • Target version set to Ready

#3 Updated by EDiGiacinto over 2 years ago

Please can you provide updated links wrt tests? all links to openQA instance are broken/not accessible to me.

Also, could you also attach examples of both xmls?

#4 Updated by pgeorgiadis over 2 years ago

The openqa instance at dede was a machine from Orthos which has been wiped since then. So, there are no data I can help you with anymore, it's been 1 year ago and I was just experimenting during hackweek. I am very sorry :/ I think what I did is just ran the basic avocado test taken from their documentation.

#5 Updated by metan over 2 years ago

This is also related to action #27726 [kernel] NVMe Over Fabrics Unit Test Framework in which we are asked to integrate nose2 based testsuite into openQA, nosetest can produce xunit as well so it would be nice if the openQA import was compatible with nose2 export too (I've started to discuss this in action #18000 but I was redirected here).

#6 Updated by okurz over 2 years ago

  • Subject changed from parse_junit_log is crashing with xunit from Avocado to parse_junit_log is crashing with xunit from Avocado / proper xunit parsing in openQA

#7 Updated by coolo over 2 years ago

  • Assignee set to EDiGiacinto

#8 Updated by EDiGiacinto about 2 years ago

  • Target version changed from Ready to Current Sprint

#9 Updated by EDiGiacinto about 2 years ago

PR: https://github.com/os-autoinst/os-autoinst/pull/910 - it will provide a general support to XUnit, but only xunit specification i could find were in https://github.com/lukejpreston/xunit-viewer/tree/master/data, that being said, it might require adaptations, or creation of specific parser sub-implementations depending on the framework output.

#10 Updated by EDiGiacinto about 2 years ago

  • Status changed from New to In Progress

#11 Updated by EDiGiacinto about 2 years ago

  • Status changed from In Progress to Resolved

Marking this as solved since now we have a more abstract interface for parsing different formats - Avocado wasn't explictly tested, but should be covered by the XUnit support we provide.

Please reopen if adaptations to the parsing of tests are needed.

#12 Updated by szarate about 2 years ago

  • Target version changed from Current Sprint to Done

Also available in: Atom PDF