action #17412
Updated by pgeorgiadis over 7 years ago
The actual test is testing if the return code from the command `zypper patch` was 0, while it should actually make sure that the maintenance update can be installed on the system. So, basically, the tests do not do anything useful if update is not part of installed base system. In matter of fact, in that case they do more harm than good, because the reviewer sees 'green' and thinks that the installation was OK -- but it might be that actual pkg installation it never happened. For example: SUSE:Maintenance:4210:128510 SUSE:Maintenance:4225:128515 ... and every other QAM update that contains packages that are not coming from the base. Simply put, if there's no package, there's no patch to be applied: > https://openqa.suse.de/tests/791585#step/install_update/28 As a result a great number of Maintenance updates look 'green' -- this is an illusion. We need to trust this tests, to rely on the results. In that direction, we would like to make sure that these tests are going to be trustworthy. By trustworthy I mean: > If there's a failed maintenance incident test, the incident should be automatically rejected by the system and submit a bug report. No review should be needed. > If there's a passed maintenance incident test, the installation/update of the incident should be considered OK. No review should be needed. > If the test sees something that it cannot decide if it's OK or NO, it should mark the test as soft-failed until a QAM guy fixes it and restarts it. Only when it's green is going to be passed. In this regard, either we trust our tests, or not. So, this is the purpose of this ticket. Here's some hard requirements from my side: 1. The test has to always check if the testing repository is empty or not. 2. The test has to make sure that all packages included into the maintenance update are available (can be found) into the testing repositories (e.g. `zypper se -s $pkg-$version --from $repo`) 3. The test has to make sure that all the packages + the correct version of those, can be installed into the SUT without errors or conflicts (zypper in) 4. Another test has to make sure that all packages can also be updated (zypper up scenario) 5. All the PASSED maintenance incidents should be trusted without a need for review. 6. openQA will never flag a faulty installation test as passed. 7. openQA will never fail a proper installation testsuite (e.g. minimal, minimal-full) because of unrelated test-modules. 8. openQA will never fail an installation test because of infrastructure problems (e.g. no space left on the device) 9. If the test fails, the update should never come to Maintenance but get rejected and file a bug (both actions automatically) without review. problems into building this: In order to know what are the supported combinations (because not all packages could be installed every time) we can use https://github.com/SUSE/connect/wiki/SCC-API-(Implemented)#packagelist-search in order to map them and afterwards to provide these information into the maintenance bot in order to trigger the test. In that way, we might have different installation scenarios for different products/extensions. e.g if there are devel packages, there should be a VM with SLES+SDK. This tool from SCC is providing 'from where' each package can be installed. If there's RPM output after the installation of a package, we should check if there's is something considered as expected/generic (e.g. systemd symlink created or destroyed, conf.rpmnew file, etc) and continue the test. But, if there's something else that we cannot say up-front: this is OK or this not OK, then the openQA test has to mark itself as a soft-failed. The QAM Engineer has to inspect the situation (this would be the only case to review a maintenance incident test) and if this OK, he has to hard-code additional RPM output as 'OK' scenario and then re-run the test. The test should should never be passed if it's not green (!), but not a fake green :P In the long-term, we would like MTUI to be able to read the results from openQA (related to installation) and export them into the our test-report. So let's start: I think the first step is to see how this is done in opensuse: [qam_verify_package_install](https://openqa.opensuse.org/tests/363103/modules/qam_verify_package_install/steps/1/src.txt) maybe we can use it or write something better. Let's see.