action #10210
closedDifferent classes of softfails, e.g. "softfail" needle
0%
Description
User story¶
As a QA tester I want to be able to create needles that make a test softfail if found to not need to change test code with a "check_screen and record_soft_failure"
acceptance criteria¶
- AC1: A test module softfails if a specified needle with a certain property fails
- AC2: The test module does not fail if the softfail needle is checked but not matching
tasks¶
- Add checkbox for property on needle editor
- Match with priority on softfail needle
- Make testmodule softfail if softfail needle matches, e.g. "record_soft_failure"
further details¶
There are a number of different 'classes' of softfails
Two obvious examples are:
Workarounds (either in terms of code or needles). ie. "openQA detected this, and did something to workaround it"
Acceptable failures ie. "openQA detected this, and we accept it as not serious, but it is a failure"
We need to handle these two conditions differently - I think we need different icons/colours (Orange and Yellow, not just yellow?) and possibly different API calls (record_soft_failure and record_workaround?).
Added by okurz: The problem is that if two needles match, one old one and one new one which is more specific and marked as "workaround", it will match the old one and not mark a softfail. What we need is a "priority match"
For motivation also see #12162
Updated by coolo about 8 years ago
I wouldn't overload the semantic of a passing test too much. A softfail is already a quite complex thingy. If acceptable failures are acceptable, then they are just that: acceptable green to openQA and the failure should live in bugzilla.
What you're talking about are expected failures. They are not softfail, they are fails - but we continue afterwards. I don't think it should be openQA's job to visualize all aspects of product quality. E.g. if there are no release notes in the product, but the specficiation says there should be some, the test failed. If it's acceptable, we continue afterwards and test other things. But the exact colour of the bubble has no meaning. If there are no release notes, all bubbles would be red right now - and orange in your scenario. But they would still all be the same and the gain is zero.
Updated by okurz almost 8 years ago
- Has duplicate action #12168: Support for "fail"/"softfail" needles added
Updated by okurz over 7 years ago
- Category set to 132
- Status changed from New to Resolved
All AC are covered with commit 733cf9f
Author: Adam Williamson awilliam@redhat.com
Date: Thu Jul 7 17:12:47 2016 -0700
when match is equal quality, prefer workaround needle
resolves openQA issue #779. When two needles both match with
the same match quality, we currently just take whichever sorts
first by name alphabetically. We would like to predictably
prefer workaround needles to non-workaround in this case, so
we rename `cmp_by_error__` to `cmp_by_error_type_` and have it
also consider the needle type and prefer workaround needles, if
the match quality is equal.
We have a proper needle editor which offers the "workaround" property to be set on a needle together with a help popup explaining that. os-autoinst matches on workaround needles with priority. What is not included is different classes for "just workaround" or "failure detected" but I never saw this need expressed in the last months. Also, we changed semantics regarding "passed"/"softfailed"/"failed" to mark a job at least as "softfailed" if any module fails or records a soft fail. So no more confusing "The job passed but what do the detailed module labels mean?"