action #17412
closed[qam] Work for Maintenance Incidents tests
100%
Description
The actual test is testing if the return code from the command zypper patch was 0, while it should actually make sure that the maintenance update can be installed on the system. So, basically, the tests do not do anything useful if update is not part of installed base system. In matter of fact, in that case they do more harm than good, because the reviewer sees 'green' and thinks that the installation was OK -- but it might be that actual pkg installation never happened.
For example:
SUSE:Maintenance:4210:128510
SUSE:Maintenance:4225:128515
... and every other QAM update that contains packages that are not coming from the base.
Simply put, if there's no package, there's no patch to be applied:
As a result a great number of Maintenance updates look 'green' -- an illusion. As a result it came to my attention that all qam-nue members misunderstood this and we though that every single pkg gets installed in openQA. We need to trust this tests, to rely on the results. So, either we have to disable minimal testing in pkgs which are not part of the minimal system or enhance it to include every pkg we test.
I think the first step is to see how this is done in opensuse: qam_verify_package_install maybe we can use it or write something better.
Updated by pgeorgiadis almost 8 years ago
- Category set to Enhancement to existing tests
- Assignee set to osukup
Updated by pgeorgiadis almost 8 years ago
- Description updated (diff)
- Private changed from No to Yes
Updated by osukup almost 8 years ago
- Priority changed from High to Low
- Private changed from Yes to No
This is QAM minimal , and as is in our docu is valid only for packages listed in https://qam.suse.de/metadata/packages-to-be-tested-on-minimal-systems
Updated by pluskalm almost 8 years ago
- Subject changed from Work for Maintenance Incidents tests to [qam] Work for Maintenance Incidents tests
Updated by osukup almost 8 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 10
Updated by osukup almost 8 years ago
- % Done changed from 10 to 50
Updated by osukup almost 8 years ago
- % Done changed from 50 to 80
Updated by osukup over 7 years ago
- Status changed from In Progress to Resolved
- % Done changed from 80 to 100
Updated by openqa_review over 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/7004427
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The label in the openQA scenario is removed
Updated by openqa_review about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/7207183
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Updated by openqa_review about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/7501338
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Updated by openqa_review almost 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/7947498
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Updated by openqa_review almost 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/8165806
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Updated by openqa_review over 2 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/8753678
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 152 days if nothing changes in this ticket.
Updated by openqa_review about 2 years ago
- Status changed from Resolved to Feedback
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/9965181
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 352 days if nothing changes in this ticket.
Updated by osukup about 2 years ago
- Status changed from Feedback to Resolved
openqa_review ? wrongly set feedback
Updated by openqa_review almost 2 years ago
- Status changed from Resolved to Feedback
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/10420761
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 92 days if nothing changes in this ticket.
Updated by livdywan almost 2 years ago
- Status changed from Feedback to In Progress
- Target version set to Ready
osukup wrote:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2563
Looks like this is back. It actually seems to fail in record_soft_failure("Patch isn't needed on minimal installation poo#17412"); (see https://openqa.suse.de/tests/10420761/modules/install_update/steps/1/src).
See also
https://gitlab.suse.de/qa-maintenance/metadata/-/blob/master/bot-ng/sles15sp4.yml#L207-266 for the definition of packages in minimal, likely openSSL needs to be specified more exactly
Updated by osukup almost 2 years ago
- Status changed from In Progress to Resolved
I thin we should forward this ticket to qem-openqa-review squad as this is exactly in their area of interest ...
https://gitlab.suse.de/qa-maintenance/metadata/-/merge_requests/798
Updated by openqa_review over 1 year ago
- Status changed from Resolved to Feedback
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/10828301
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by osukup over 1 year ago
- Status changed from Feedback to Resolved
-- after check --> incident is in Storage 7 repo, so not needed on minimal installation.
Updated by openqa_review over 1 year ago
- Status changed from Resolved to Feedback
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal@s390x-zVM-vswitch-l2
https://openqa.suse.de/tests/10933525
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by openqa_review over 1 year ago
- Status changed from Resolved to Feedback
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/11147781
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by okurz over 1 year ago
@osukup https://openqa.suse.de/tests/11147781#step/install_update/83 uses this progress ticket in a soft-fail reference. I suggest to remove the soft-failure reference, it can be a "record_info" or simple comment in the test code