Project

General

Profile

Actions

action #70915

closed

Bugref comments carried over even if status changed failed -> incomplete

Added by favogt about 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
QA - future
Start date:
2020-09-03
Due date:
% Done:

0%

Estimated time:

Description

See https://openqa.opensuse.org/tests/1381130#next_previous

The comment from the last failing test was carried over to the current incomplete one as the set of failing modules is equal.
However, in the case of the incomplete test some modules were not executed at all, so their state is unknown at that point.


Related issues 1 (0 open1 closed)

Related to openQA Project - action #94880: Carry-over despite mismatch of failing job modulesResolvedmkittler2021-06-29

Actions
Actions #1

Updated by okurz about 4 years ago

  • Category set to Feature requests
  • Priority changed from Normal to Low
  • Target version set to future

it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same. May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?

Actions #2

Updated by favogt about 4 years ago

  • Category deleted (Feature requests)
  • Priority changed from Low to Normal
  • Target version deleted (future)

it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same.

Ouch. That means that any "@ttm ignore" comment with a bugref is a ticking time bomb?

May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?

The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.

Actions #3

Updated by okurz about 4 years ago

  • Category set to Feature requests
  • Priority changed from Normal to Low
  • Target version set to future

favogt wrote:

it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same.

Ouch. That means that any "@ttm ignore" comment with a bugref is a ticking time bomb?

yes

May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?

The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.

That's nothing new because human reviewers also ignore new problems when they just look at the failed module name from the test overview page

Actions #4

Updated by favogt about 4 years ago

  • Category deleted (Feature requests)
  • Priority changed from Low to Normal

okurz wrote:

favogt wrote:

The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.

That's nothing new because human reviewers also ignore new problems when they just look at the failed module name from the test overview page

They definitely don't if the new test incompleted. Incompletes are clearly distinguishable from other failures and except for rare circumstances always show a bigger issue. Even if the color change is somehow missed, human reviewers have to visit the job page and look at it to even navigate to the page to comment "@ttm ignore".

Actions #5

Updated by okurz about 4 years ago

  • Category set to Feature requests

why do you keep changing the fields without explaining why? At least I don't understand but I think because you kept the target version it was not by mistake now. I regard this as "feature request" as described in #70915#note-1 . And the target version is "future" meaning that currently the SUSE QA tools team does not plan this. If you want to keep prio higher than normal then this is fine.

Actions #6

Updated by okurz about 4 years ago

If you want to help get this ticket done but not write the code yourself you can help to use the template https://progress.opensuse.org/projects/openqav3/wiki/#Feature-requests which is making it easier for any contributor to clearly understand what is expected and what is the goal. As you probably see it would be hard to regard this as a defect following https://progress.opensuse.org/projects/openqav3/wiki/#Defects which I think would be harder to fill because there is no "hypothesis" what could have caused this "regression" when it is not a regression.

Actions #7

Updated by favogt about 4 years ago

okurz wrote:

why do you keep changing the fields without explaining why?

That's my way of asking for a reevaluation.

At least I don't understand but I think because you kept the target version it was not by mistake now. I regard this as "feature request" as described in #70915#note-1 . And the target version is "future" meaning that currently the SUSE QA tools team does not plan this. If you want to keep prio higher than normal then this is fine.

I absolutely view this as a critical bug and not a missing feature, see my comments. It does not have to be a regression to be a bug.

Actions #8

Updated by okurz about 4 years ago

favogt wrote:

That's my way of asking for a reevaluation.

ok, got it.

I absolutely view this as a critical bug and not a missing feature, see my comments. It does not have to be a regression to be a bug.

You can see it this way but I am keeping in mind how we can best describe tickets so that someone can work on this. And for this it is important to know if it's a regression.

Actions #9

Updated by mkittler over 3 years ago

  • Assignee set to mkittler

This issue came up in the context of #94880. I belief the fix is rather simple so I've been creating a PR, see https://github.com/os-autoinst/openQA/pull/3991 (and in particular https://github.com/os-autoinst/openQA/pull/3991/commits/36ebb8eb008c77edc865fa2bb4df08946fe9fb01).

Actions #10

Updated by mkittler over 3 years ago

  • Related to action #94880: Carry-over despite mismatch of failing job modules added
Actions #11

Updated by mkittler over 3 years ago

  • Status changed from New to In Progress
Actions #12

Updated by mkittler over 3 years ago

  • Status changed from In Progress to Feedback

PR has been merged

Actions #13

Updated by mkittler over 3 years ago

  • Status changed from Feedback to Resolved

The PR is deployed in production. The change has a unit test and works very likely so I wouldn't take the effort of reproducing the case in production. If it doesn't work after all we can still re-open the ticket.

Actions

Also available in: Atom PDF