Project

General

Profile

action #104568

[qe-core] QAM maintenance tests: openQA to be able to test updates after some of them were rejected

Added by vpelcak 5 months ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Enhancement to existing tests
Target version:
-
Start date:
2022-01-03
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

When testing of maintenance update identifies broken update and it is rejected, the tests will start failing as the repository will become empty.
That kills the testing for the day.

Acceptance criteria

  • AC1: When the update is rejected, restarted tests will proceed like nothing happened and will test the remaining maintenance updates.

Suggestions

  • Make openQA smarter itself to identify this kind of situation
  • Modify the initial tests to gracefully skip empty repositories

History

#1 Updated by vpelcak 5 months ago

  • Subject changed from Add ability of openQA to test updates after some of them were rejected to openQA to be able to test updates after some of them were rejected

#2 Updated by okurz 5 months ago

  • Project changed from QA to openQA Tests
  • Subject changed from openQA to be able to test updates after some of them were rejected to [qe-core] QAM maintenance tests: openQA to be able to test updates after some of them were rejected
  • Category set to Enhancement to existing tests

This is actually something that can be solved just within os-autoinst-distri-opensuse, so changing project scope accordingly. Can you provide an example of a failing test which hit such a case? I am pretty sure that in whatever test module such tests fail the tests can be adapted. Do you think there is anything speaking against simply ignoring empty repositories? So instead of trying to install packages from a repository and fail simply don't treat it as an error, either by looking for installable packages and skipping if there are no packages or catch the error and skip.

I guess this needs a change in one of the following locations:

#3 Updated by vpelcak 5 months ago

I am absolutely alright with whatever solution as long as it works reliably and doesn't produce false positives and negatives.

#4 Updated by hurhaj 4 months ago

vpelcak wrote:

I am absolutely alright with whatever solution as long as it works reliably and doesn't produce false positives and negatives.

Empty repo can happen, for example by mistake of maint coordinator (as I saw recently). It's much less likely than the scenario of rejection and it can also be detected by maint team themselves, QEM when we see empty update, etc. I believe there's no need to check for every possible issue on every possible step of the process. Therefore I believe Oliver's proposal is absolutely fine.

#5 Updated by osukup 4 months ago

okurz wrote:

This is actually something that can be solved just within os-autoinst-distri-opensuse, so changing project scope accordingly. Can you provide an example of a failing test which hit such a case? I am pretty sure that in whatever test module such tests fail the tests can be adapted. Do you think there is anything speaking against simply ignoring empty repositories? So instead of trying to install packages from a repository and fail simply don't treat it as an error, either by looking for installable packages and skipping if there are no packages or catch the error and skip.

I guess this needs a change in one of the following locations:

I agree with okurz .. it can be solved at test level adding common test if URL in MAINT_TEST_REPO/INCIDENT_REPO exist and then change vars on the fly or even better in main.pm on starting sequence of test

Also available in: Atom PDF