action #70915
closedBugref comments carried over even if status changed failed -> incomplete
0%
Description
See https://openqa.opensuse.org/tests/1381130#next_previous
The comment from the last failing test was carried over to the current incomplete one as the set of failing modules is equal.
However, in the case of the incomplete test some modules were not executed at all, so their state is unknown at that point.
Updated by okurz over 4 years ago
- Category set to Feature requests
- Priority changed from Normal to Low
- Target version set to future
it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same. May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?
Updated by favogt over 4 years ago
- Category deleted (
Feature requests) - Priority changed from Low to Normal
- Target version deleted (
future)
it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same.
Ouch. That means that any "@ttm ignore" comment with a bugref is a ticking time bomb?
May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?
The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.
Updated by okurz over 4 years ago
- Category set to Feature requests
- Priority changed from Normal to Low
- Target version set to future
favogt wrote:
it's certainly a correct observation but not a regression. The carry over was always working on the module level and actually regardless of the overall job status. It just looks if all previous modules passed and if the first failing module is the same.
Ouch. That means that any "@ttm ignore" comment with a bugref is a ticking time bomb?
yes
May I assume that the impact on you is minor, you just wanted to report about the issue as you see the behaviour as unexpected?
The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.
That's nothing new because human reviewers also ignore new problems when they just look at the failed module name from the test overview page
Updated by favogt over 4 years ago
- Category deleted (
Feature requests) - Priority changed from Low to Normal
okurz wrote:
favogt wrote:
The impact is potentially catastrophic, as this results in openQA ignoring test module failures. That's the worst case scenario.
That's nothing new because human reviewers also ignore new problems when they just look at the failed module name from the test overview page
They definitely don't if the new test incompleted. Incompletes are clearly distinguishable from other failures and except for rare circumstances always show a bigger issue. Even if the color change is somehow missed, human reviewers have to visit the job page and look at it to even navigate to the page to comment "@ttm ignore".
Updated by okurz over 4 years ago
- Category set to Feature requests
why do you keep changing the fields without explaining why? At least I don't understand but I think because you kept the target version it was not by mistake now. I regard this as "feature request" as described in #70915#note-1 . And the target version is "future" meaning that currently the SUSE QA tools team does not plan this. If you want to keep prio higher than normal then this is fine.
Updated by okurz over 4 years ago
If you want to help get this ticket done but not write the code yourself you can help to use the template https://progress.opensuse.org/projects/openqav3/wiki/#Feature-requests which is making it easier for any contributor to clearly understand what is expected and what is the goal. As you probably see it would be hard to regard this as a defect following https://progress.opensuse.org/projects/openqav3/wiki/#Defects which I think would be harder to fill because there is no "hypothesis" what could have caused this "regression" when it is not a regression.
Updated by favogt over 4 years ago
okurz wrote:
why do you keep changing the fields without explaining why?
That's my way of asking for a reevaluation.
At least I don't understand but I think because you kept the target version it was not by mistake now. I regard this as "feature request" as described in #70915#note-1 . And the target version is "future" meaning that currently the SUSE QA tools team does not plan this. If you want to keep prio higher than normal then this is fine.
I absolutely view this as a critical bug and not a missing feature, see my comments. It does not have to be a regression to be a bug.
Updated by okurz over 4 years ago
favogt wrote:
That's my way of asking for a reevaluation.
ok, got it.
I absolutely view this as a critical bug and not a missing feature, see my comments. It does not have to be a regression to be a bug.
You can see it this way but I am keeping in mind how we can best describe tickets so that someone can work on this. And for this it is important to know if it's a regression.
Updated by mkittler over 3 years ago
- Assignee set to mkittler
This issue came up in the context of #94880. I belief the fix is rather simple so I've been creating a PR, see https://github.com/os-autoinst/openQA/pull/3991 (and in particular https://github.com/os-autoinst/openQA/pull/3991/commits/36ebb8eb008c77edc865fa2bb4df08946fe9fb01).
Updated by mkittler over 3 years ago
- Related to action #94880: Carry-over despite mismatch of failing job modules added
Updated by mkittler over 3 years ago
- Status changed from Feedback to Resolved
The PR is deployed in production. The change has a unit test and works very likely so I wouldn't take the effort of reproducing the case in production. If it doesn't work after all we can still re-open the ticket.