Project

General

Profile

Actions
Copy link

action #104568

open

[qe-core] QAM maintenance tests: openQA to be able to test updates after some of them were rejected

Added by vpelcak about 3 years ago. Updated 12 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Enhancement to existing tests
Target version:
-
Start date:
2022-01-03
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

When testing of maintenance update identifies broken update and it is rejected, the tests will start failing as the repository will become empty.
That kills the testing for the day.

Acceptance criteria

  • AC1: When the update is rejected, restarted tests will proceed like nothing happened and will test the remaining maintenance updates.

Suggestions

  • Make openQA smarter itself to identify this kind of situation
  • Modify the initial tests to gracefully skip empty repositories
Actions
Copy link
 #1

Updated by vpelcak about 3 years ago

  • Subject changed from Add ability of openQA to test updates after some of them were rejected to openQA to be able to test updates after some of them were rejected
Actions
Copy link
 #2

Updated by okurz about 3 years ago

  • Project changed from QA (public) to openQA Tests (public)
  • Subject changed from openQA to be able to test updates after some of them were rejected to [qe-core] QAM maintenance tests: openQA to be able to test updates after some of them were rejected
  • Category set to Enhancement to existing tests

This is actually something that can be solved just within os-autoinst-distri-opensuse, so changing project scope accordingly. Can you provide an example of a failing test which hit such a case? I am pretty sure that in whatever test module such tests fail the tests can be adapted. Do you think there is anything speaking against simply ignoring empty repositories? So instead of trying to install packages from a repository and fail simply don't treat it as an error, either by looking for installable packages and skipping if there are no packages or catch the error and skip.

I guess this needs a change in one of the following locations:

Actions
Copy link
 #3

Updated by vpelcak about 3 years ago

I am absolutely alright with whatever solution as long as it works reliably and doesn't produce false positives and negatives.

Actions
Copy link
 #4

Updated by hurhaj about 3 years ago

vpelcak wrote:

I am absolutely alright with whatever solution as long as it works reliably and doesn't produce false positives and negatives.

Empty repo can happen, for example by mistake of maint coordinator (as I saw recently). It's much less likely than the scenario of rejection and it can also be detected by maint team themselves, QEM when we see empty update, etc. I believe there's no need to check for every possible issue on every possible step of the process. Therefore I believe Oliver's proposal is absolutely fine.

Actions
Copy link
 #5

Updated by osukup about 3 years ago

okurz wrote:

This is actually something that can be solved just within os-autoinst-distri-opensuse, so changing project scope accordingly. Can you provide an example of a failing test which hit such a case? I am pretty sure that in whatever test module such tests fail the tests can be adapted. Do you think there is anything speaking against simply ignoring empty repositories? So instead of trying to install packages from a repository and fail simply don't treat it as an error, either by looking for installable packages and skipping if there are no packages or catch the error and skip.

I guess this needs a change in one of the following locations:

I agree with @okurz .. it can be solved at test level adding common test if URL in MAINT_TEST_REPO/INCIDENT_REPO exist and then change vars on the fly or even better in main.pm on starting sequence of test

Actions
Copy link
 #6

Updated by slo-gin about 2 years ago

This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.

Actions
Copy link
 #7

Updated by slo-gin 12 months ago

This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.

Actions
Copy link

Also available in: Atom PDF