Project

General

Profile

action #68413

De-duplicate definitions of needle soft-fail markers (was: screenshot shows soft-fail border for a needle with bug reference but the module turns out to be "passed", expected "softfailed")

Added by okurz 10 months ago. Updated 6 months ago.

Status:
Workable
Priority:
Normal
Assignee:
-
Category:
Concrete Bugs
Target version:
Start date:
2020-06-24
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

see https://openqa.opensuse.org/tests/1306944#step/wine/26

Reproducible

Reproduced, e.g. in the same scenario in other jobs the same

Expected result

The softfailed needle should make the module result turn "softfailed" as well

Problem

Likely a regression that was introduced within the last year with some changes regarding the handling of softfails

Further details

Always latest result in this scenario: latest

History

#1 Updated by okurz 10 months ago

  • Project changed from openQA Tests to openQA Project
  • Category changed from Bugs in existing tests to Concrete Bugs
  • Status changed from New to Workable
  • Assignee set to Xiaojing_liu
  • Target version set to Ready

@Xiaojing_liu can you please read the ticket and state if you can explain what is happening here? Could this be related to work done by you some months ago about handling of softfailed/workaround needles?

#2 Updated by Xiaojing_liu 10 months ago

  • Status changed from Workable to In Progress

I check this issue, it is related to the pr that I worked some months ago. I did not consider the needle file contained boo.
Made a pull request to fix this: https://github.com/os-autoinst/os-autoinst/pull/1450

#3 Updated by okurz 10 months ago

The PR is merged however I think the reason why we have this problem in the first place was os-autoinst commit ed3da7cc that introduced the regex /\S+\-(bsc|poo|bnc)(\d+)\-\S+/ within os-autoinst which we already have in openQA. We should try to find a way that we only have one place for the definition. E.g. we should be able to move the corresponding code from openQA to os-autoinst and just use it from there.

#4 Updated by Xiaojing_liu 9 months ago

okurz wrote:

The PR is merged however I think the reason why we have this problem in the first place was os-autoinst commit ed3da7cc that introduced the regex /\S+\-(bsc|poo|bnc)(\d+)\-\S+/ within os-autoinst which we already have in openQA. We should try to find a way that we only have one place for the definition. E.g. we should be able to move the corresponding code from openQA to os-autoinst and just use it from there.

I could not find a way to define this in one place. Both os-autoinst and openQA need to parse the needle file name.

#5 Updated by Xiaojing_liu 9 months ago

  • Status changed from In Progress to Feedback
  • Assignee deleted (Xiaojing_liu)

I did not find a way to improve the code, so I delete the Assignee and set it to Feedback.

#6 Updated by okurz 9 months ago

  • Subject changed from screenshot shows soft-fail border for a needle with bug reference but the module turns out to be "passed", expected "softfailed" to De-duplicate definitions of needle soft-fail markers (was: screenshot shows soft-fail border for a needle with bug reference but the module turns out to be "passed", expected "softfailed")
  • Status changed from Feedback to Workable

Ok, clear. Let's still try to use this ticket to address this issue

#7 Updated by mkittler 8 months ago

If the needle is marked as workaround needle¹, its match should be recorded as softfail (and the module result set to softfailed) by os-autoinst - regardless whether the bugref is recognized or not. Judging by the Git history of the needles repo I'm wondering whether the problem was that the needle has not been marked as workaround needle from the beginning. However, then it is not clear to me why the workaround property is present within details-wine.json of the mentioned job.

Regarding the duplication: We could create a Perl module which provides the regex. We could make it part of os-autoinst and install it into the global Perl prefix or we make it part of the new repository https://github.com/os-autoinst/os-autoinst-common and use it via Git subrepo.


¹ "properties": ["workaround"] appears in the JSON file

#8 Updated by okurz 7 months ago

mkittler wrote:

[…] I'm wondering whether the problem was that the needle has not been marked as workaround needle from the beginning

Because a needle is also regarded as "workaround" based on a ticket reference in the needle's filename without the property being set.

#9 Updated by okurz 6 months ago

  • Target version changed from Ready to future

Also available in: Atom PDF