Project

General

Profile

action #70915

Bugref comments carried over even if status changed failed -> incomplete

Added by favogt 8 months ago. Updated 7 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2020-09-03
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

See https://openqa.opensuse.org/tests/1381130#next_previous

The comment from the last failing test was carried over to the current incomplete one as the set of failing modules is equal.
However, in the case of the incomplete test some modules were not executed at all, so their state is unknown at that point.

History

#1 Updated by okurz 7 months ago

  • Category set to Feature requests
  • Priority changed from Normal to Low
  • Target version set to future

it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same. May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?

#2 Updated by favogt 7 months ago

  • Category deleted (Feature requests)
  • Priority changed from Low to Normal
  • Target version deleted (future)

it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same.

Ouch. That means that any "@ttm ignore" comment with a bugref is a ticking time bomb?

May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?

The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.

#3 Updated by okurz 7 months ago

  • Category set to Feature requests
  • Priority changed from Normal to Low
  • Target version set to future

favogt wrote:

it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same.

Ouch. That means that any "@ttm ignore" comment with a bugref is a ticking time bomb?

yes

May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?

The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.

That's nothing new because human reviewers also ignore new problems when they just look at the failed module name from the test overview page

#4 Updated by favogt 7 months ago

  • Category deleted (Feature requests)
  • Priority changed from Low to Normal

okurz wrote:

favogt wrote:

The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.

That's nothing new because human reviewers also ignore new problems when they just look at the failed module name from the test overview page

They definitely don't if the new test incompleted. Incompletes are clearly distinguishable from other failures and except for rare circumstances always show a bigger issue. Even if the color change is somehow missed, human reviewers have to visit the job page and look at it to even navigate to the page to comment "@ttm ignore".

#5 Updated by okurz 7 months ago

  • Category set to Feature requests

why do you keep changing the fields without explaining why? At least I don't understand but I think because you kept the target version it was not by mistake now. I regard this as "feature request" as described in #70915#note-1 . And the target version is "future" meaning that currently the SUSE QA tools team does not plan this. If you want to keep prio higher than normal then this is fine.

#6 Updated by okurz 7 months ago

If you want to help get this ticket done but not write the code yourself you can help to use the template https://progress.opensuse.org/projects/openqav3/wiki/#Feature-requests which is making it easier for any contributor to clearly understand what is expected and what is the goal. As you probably see it would be hard to regard this as a defect following https://progress.opensuse.org/projects/openqav3/wiki/#Defects which I think would be harder to fill because there is no "hypothesis" what could have caused this "regression" when it is not a regression.

#7 Updated by favogt 7 months ago

okurz wrote:

why do you keep changing the fields without explaining why?

That's my way of asking for a reevaluation.

At least I don't understand but I think because you kept the target version it was not by mistake now. I regard this as "feature request" as described in #70915#note-1 . And the target version is "future" meaning that currently the SUSE QA tools team does not plan this. If you want to keep prio higher than normal then this is fine.

I absolutely view this as a critical bug and not a missing feature, see my comments. It does not have to be a regression to be a bug.

#8 Updated by okurz 7 months ago

favogt wrote:

That's my way of asking for a reevaluation.

ok, got it.

I absolutely view this as a critical bug and not a missing feature, see my comments. It does not have to be a regression to be a bug.

You can see it this way but I am keeping in mind how we can best describe tickets so that someone can work on this. And for this it is important to know if it's a regression.

Also available in: Atom PDF