Project

General

Profile

Actions
Copy link

action #17412

closed

[qam] Work for Maintenance Incidents tests

Added by pgeorgiadis almost 8 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Low
Assignee:
osukup
Category:
Enhancement to existing tests
Target version:
openQA Project (public) - Ready
Start date:
2017-03-01
Due date:
% Done:

100%

Estimated time:
Difficulty:

Description

The actual test is testing if the return code from the command zypper patch was 0, while it should actually make sure that the maintenance update can be installed on the system. So, basically, the tests do not do anything useful if update is not part of installed base system. In matter of fact, in that case they do more harm than good, because the reviewer sees 'green' and thinks that the installation was OK -- but it might be that actual pkg installation never happened.

For example:
SUSE:Maintenance:4210:128510
SUSE:Maintenance:4225:128515
... and every other QAM update that contains packages that are not coming from the base.

Simply put, if there's no package, there's no patch to be applied:

https://openqa.suse.de/tests/791585#step/install_update/28

As a result a great number of Maintenance updates look 'green' -- an illusion. As a result it came to my attention that all qam-nue members misunderstood this and we though that every single pkg gets installed in openQA. We need to trust this tests, to rely on the results. So, either we have to disable minimal testing in pkgs which are not part of the minimal system or enhance it to include every pkg we test.

I think the first step is to see how this is done in opensuse: qam_verify_package_install maybe we can use it or write something better.

Actions
Copy link

Also available in: Atom PDF