Project

General

Profile

Actions
Copy link

action #17412

closed

[qam] Work for Maintenance Incidents tests

Added by pgeorgiadis about 7 years ago. Updated 11 months ago.

Status:
Resolved
Priority:
Low
Assignee:
osukup
Category:
Enhancement to existing tests
Target version:
openQA Project - Ready
Start date:
2017-03-01
Due date:
% Done:

100%

Estimated time:
Difficulty:

Description

The actual test is testing if the return code from the command zypper patch was 0, while it should actually make sure that the maintenance update can be installed on the system. So, basically, the tests do not do anything useful if update is not part of installed base system. In matter of fact, in that case they do more harm than good, because the reviewer sees 'green' and thinks that the installation was OK -- but it might be that actual pkg installation never happened.

For example:
SUSE:Maintenance:4210:128510
SUSE:Maintenance:4225:128515
... and every other QAM update that contains packages that are not coming from the base.

Simply put, if there's no package, there's no patch to be applied:

https://openqa.suse.de/tests/791585#step/install_update/28

As a result a great number of Maintenance updates look 'green' -- an illusion. As a result it came to my attention that all qam-nue members misunderstood this and we though that every single pkg gets installed in openQA. We need to trust this tests, to rely on the results. So, either we have to disable minimal testing in pkgs which are not part of the minimal system or enhance it to include every pkg we test.

I think the first step is to see how this is done in opensuse: qam_verify_package_install maybe we can use it or write something better.

Actions
Copy link
 #1

Updated by pgeorgiadis about 7 years ago

  • Category set to Enhancement to existing tests
  • Assignee set to osukup
Actions
Copy link
 #2

Updated by pgeorgiadis about 7 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by pgeorgiadis about 7 years ago

  • Description updated (diff)
  • Private changed from No to Yes
Actions
Copy link
 #4

Updated by osukup about 7 years ago

  • Priority changed from High to Low
  • Private changed from Yes to No

This is QAM minimal , and as is in our docu is valid only for packages listed in https://qam.suse.de/metadata/packages-to-be-tested-on-minimal-systems

Actions
Copy link
 #5

Updated by pluskalm about 7 years ago

  • Subject changed from Work for Maintenance Incidents tests to [qam] Work for Maintenance Incidents tests
Actions
Copy link
 #6

Updated by osukup about 7 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10
Actions
Copy link
 #9

Updated by osukup about 7 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 80 to 100
Actions
Copy link
 #10

Updated by openqa_review over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/7004427

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The label in the openQA scenario is removed
Actions
Copy link
 #11

Updated by openqa_review over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/7207183

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234
Actions
Copy link
 #12

Updated by openqa_review over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/7501338

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234
Actions
Copy link
 #13

Updated by openqa_review over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/7947498

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234
Actions
Copy link
 #14

Updated by openqa_review about 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/8165806

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234
Actions
Copy link
 #15

Updated by openqa_review almost 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/8753678

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 152 days if nothing changes in this ticket.

Actions
Copy link
 #16

Updated by openqa_review over 1 year ago

  • Status changed from Resolved to Feedback

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/9965181

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 352 days if nothing changes in this ticket.

Actions
Copy link
 #17

Updated by osukup over 1 year ago

  • Status changed from Feedback to Resolved

openqa_review ? wrongly set feedback

Actions
Copy link
 #18

Updated by openqa_review over 1 year ago

  • Status changed from Resolved to Feedback

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/10420761

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 92 days if nothing changes in this ticket.

Actions
Copy link
 #19

Updated by livdywan about 1 year ago

  • Status changed from Feedback to In Progress
  • Target version set to Ready

osukup wrote:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2563

Looks like this is back. It actually seems to fail in record_soft_failure("Patch isn't needed on minimal installation poo#17412"); (see https://openqa.suse.de/tests/10420761/modules/install_update/steps/1/src).

See also
https://gitlab.suse.de/qa-maintenance/metadata/-/blob/master/bot-ng/sles15sp4.yml#L207-266 for the definition of packages in minimal, likely openSSL needs to be specified more exactly

Actions
Copy link
 #20

Updated by osukup about 1 year ago

  • Status changed from In Progress to Resolved

I thin we should forward this ticket to qem-openqa-review squad as this is exactly in their area of interest ...

https://gitlab.suse.de/qa-maintenance/metadata/-/merge_requests/798

Actions
Copy link
 #21

Updated by openqa_review about 1 year ago

  • Status changed from Resolved to Feedback

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/10828301

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions
Copy link
 #22

Updated by osukup about 1 year ago

  • Status changed from Feedback to Resolved

-- after check --> incident is in Storage 7 repo, so not needed on minimal installation.

Actions
Copy link
 #23

Updated by openqa_review about 1 year ago

  • Status changed from Resolved to Feedback

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal@s390x-zVM-vswitch-l2
https://openqa.suse.de/tests/10933525

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions
Copy link
 #24

Updated by osukup about 1 year ago

  • Status changed from Feedback to Resolved

wontfix ...

Actions
Copy link
 #25

Updated by openqa_review 12 months ago

  • Status changed from Resolved to Feedback

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: qam-minimal
https://openqa.suse.de/tests/11147781

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions
Copy link
 #26

Updated by okurz 12 months ago

@osukup https://openqa.suse.de/tests/11147781#step/install_update/83 uses this progress ticket in a soft-fail reference. I suggest to remove the soft-failure reference, it can be a "record_info" or simple comment in the test code

Actions
Copy link
 #27

Updated by osukup 11 months ago

  • Status changed from Feedback to Resolved

demoted to record_info

Actions
Copy link

Also available in: Atom PDF