Project

General

Profile

Actions

action #10210

closed

Different classes of softfails, e.g. "softfail" needle

Added by RBrownSUSE about 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
-
Start date:
2016-01-13
Due date:
% Done:

0%

Estimated time:

Description

User story

As a QA tester I want to be able to create needles that make a test softfail if found to not need to change test code with a "check_screen and record_soft_failure"

acceptance criteria

  • AC1: A test module softfails if a specified needle with a certain property fails
  • AC2: The test module does not fail if the softfail needle is checked but not matching

tasks

  • Add checkbox for property on needle editor
  • Match with priority on softfail needle
  • Make testmodule softfail if softfail needle matches, e.g. "record_soft_failure"

further details

There are a number of different 'classes' of softfails

Two obvious examples are:

Workarounds (either in terms of code or needles). ie. "openQA detected this, and did something to workaround it"

Acceptable failures ie. "openQA detected this, and we accept it as not serious, but it is a failure"

We need to handle these two conditions differently - I think we need different icons/colours (Orange and Yellow, not just yellow?) and possibly different API calls (record_soft_failure and record_workaround?).
Added by okurz: The problem is that if two needles match, one old one and one new one which is more specific and marked as "workaround", it will match the old one and not mark a softfail. What we need is a "priority match"

For motivation also see #12162


Related issues 1 (0 open1 closed)

Has duplicate openQA Project - action #12168: Support for "fail"/"softfail" needlesRejected2016-05-31

Actions
Actions #1

Updated by coolo about 8 years ago

I wouldn't overload the semantic of a passing test too much. A softfail is already a quite complex thingy. If acceptable failures are acceptable, then they are just that: acceptable green to openQA and the failure should live in bugzilla.

What you're talking about are expected failures. They are not softfail, they are fails - but we continue afterwards. I don't think it should be openQA's job to visualize all aspects of product quality. E.g. if there are no release notes in the product, but the specficiation says there should be some, the test failed. If it's acceptable, we continue afterwards and test other things. But the exact colour of the bubble has no meaning. If there are no release notes, all bubbles would be red right now - and orange in your scenario. But they would still all be the same and the gain is zero.

Actions #2

Updated by okurz almost 8 years ago

  • Subject changed from Different classes of softfails to Different classes of softfails, e.g. "softfail" needle
  • Description updated (diff)

merged in content from #12168

Actions #3

Updated by okurz almost 8 years ago

  • Has duplicate action #12168: Support for "fail"/"softfail" needles added
Actions #4

Updated by okurz over 7 years ago

  • Category set to 132
  • Status changed from New to Resolved

All AC are covered with commit 733cf9f
Author: Adam Williamson awilliam@redhat.com
Date: Thu Jul 7 17:12:47 2016 -0700

when match is equal quality, prefer workaround needle

resolves openQA issue #779. When two needles both match with
the same match quality, we currently just take whichever sorts
first by name alphabetically. We would like to predictably
prefer workaround needles to non-workaround in this case, so
we rename `cmp_by_error__` to `cmp_by_error_type_` and have it
also consider the needle type and prefer workaround needles, if
the match quality is equal.

We have a proper needle editor which offers the "workaround" property to be set on a needle together with a help popup explaining that. os-autoinst matches on workaround needles with priority. What is not included is different classes for "just workaround" or "failure detected" but I never saw this need expressed in the last months. Also, we changed semantics regarding "passed"/"softfailed"/"failed" to mark a job at least as "softfailed" if any module fails or records a soft fail. So no more confusing "The job passed but what do the detailed module labels mean?"

Actions

Also available in: Atom PDF