QA » openQA Project » openQA Tests

We don't want to paint it green

@szarate Not sure what you want to tell with your statement. Of course I am with you to not just mark "known bugs" by record_info instead of a soft-fail. Have you followed https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6589#issuecomment-456419001 and the whole discussion after that? The problem was with "noteworthy conditions" that can not have a workable bug behind where people also added record_soft_failure without bug references. Do you consider this feature worthwhile to follow up with or would you prefer an alternative?

As I was asked for further details, let me try to explain: With serial_failure_detection I mean the logic called in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6589/files#diff-45750ef51e5e8360e7ef27b51599ba4b396c661c6ef82d1fd76cecd3ebec195cR27 and the reason why we created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6589 should explain the idea as well: soft-fail also according to http://open.qa/docs/ means "known, non-critical issue", so it should only be used if we have a bugref. only this way we can ensure it is "known". Otherwise jobs will end up all yellow but no one can do anything about it, leading to alarm-fatigue

So, to clarify: the code in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6589/files#diff-45750ef51e5e8360e7ef27b51599ba4b396c661c6ef82d1fd76cecd3ebec195cR27 and https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6589/files only changes what is being pushed to serial failures. serial failures is then passed to the backend (basetest.pm) via main.pm, and it's used to fail or softfail according to it's type attribute.

In my understanding, what is being asked is to add a third type option - apart from the currently allowed soft and hard - and then in the backend where the actual parsing, regex matching and (soft)failing takes place, account for that third type using record_info to notify users and then proceeding without failing or softfailing.

Is that understanding correct?

VANASTASIADIS wrote:

In my understanding, what is being asked is to add a third type option - apart from the currently allowed soft and hard - and then in the backend where the actual parsing, regex matching and (soft)failing takes place, account for that third type using record_info to notify users and then proceeding without failing or softfailing.

Is that understanding correct?

that sounds like a good idea, yes.

Given that within the tools team we have a different focus for now and are exceeding our WIP limit I suggest you unassign again. Can you?

Apparently there was no comment here following my chat with Bill:

• There's the implementation in basetest.pm in os-autoinst which Bill would have to tackle from a Tools pov
• There's the distri part where the desired behavior can be implemented (qe-core)
• Why do we distinguish hardhard vs. softfail vs. "not a failure, not a bug" the point of that third, new option seems unclear
• We need a category that only provides more data for debugging even if by default it will not be used/needed

thank you!