Project

General

Profile

coordination #76960

Test modules that depends on PackageHub for software provided in supported SLE modules

Added by SLindoMansilla about 1 year ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Spike/Research
Target version:
-
Start date:
2020-11-04
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

  • Goal: Reduce human effort reviewing failed jobs
  • Problem: PackageHub is not available until BETA1, which causes non-sporadic failing test modules

Proposals

keep them failing during Alpha phase

  • Pros: Take over bot can label consequent failures without human intervention once a job is labeled
  • Contra:
    • Red color during alpha phase for known bug
    • Software from supported SLE modules is not tested during alpha phase

make them soft-failing during alpha phase and failing since BETA1 phase if PHub is still not available/working

  • Pros: Non-red (orange) color during alpha phase for known bug
  • Contra:
    • There is no clean way to detect in a test module if we are in alpha phase yet
    • Software from supported SLE modules is not tested during alpha phase

change test modules to not depend on PackageHub

  • Pros: Proper testing of supported software during alpha phase
  • Contra: Implementation requires big human effort

History

#1 Updated by SLindoMansilla about 1 year ago

  • Description updated (diff)

#2 Updated by SLindoMansilla about 1 year ago

  • Description updated (diff)

#3 Updated by okurz about 1 year ago

SLindoMansilla wrote:

  • Contra: There is no clean way to detect in a test module if we are in alpha phase yet

I think there is a very clean way: if get_var('BETA') …

The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.

  • Software from supported SLE modules is not tested during alpha phase

can you clarify that

#5 Updated by SLindoMansilla about 1 year ago

okurz wrote:

SLindoMansilla wrote:

  • Contra: There is no clean way to detect in a test module if we are in alpha phase yet

I think there is a very clean way: if get_var('BETA') …

GMC phase?

The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.

  • Software from supported SLE modules is not tested during alpha phase

can you clarify that

sssd test is not working because it fails trying to enable Package Hub.

#6 Updated by szarate about 1 year ago

SLindoMansilla wrote:

okurz wrote:

SLindoMansilla wrote:

  • Contra: There is no clean way to detect in a test module if we are in alpha phase yet

I think there is a very clean way: if get_var('BETA') …

GMC phase?

The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.

  • Software from supported SLE modules is not tested during alpha phase

can you clarify that

sssd test is not working because it fails trying to enable Package Hub.

To clarify further: sssd tests depend on python-pam, and for instance, machinery is enabled (but that's a phub project), we might have other tests that do depend on phub aswell...

#7 Updated by jlausuch about 1 year ago

When is BETA variable supposed to be removed?

#8 Updated by SLindoMansilla about 1 year ago

jlausuch wrote:

When is BETA variable supposed to be removed?

With GMC (Gold Master Candidate). But :), when we have the candidate build, we actually have a GMC Candidate, also GMCC, which means that is BETA yet, and therefore it still needs the beta handling (beta popup), after the GMCC is declared GMC, the beta popup disappears and we can remove the beta flag. But :), if we have two versions in the tests, e.g. 15-SP3, 12-SP6, it has to be set in YAML schedule per product, because they are not going to be GMC nor alpha at the same time.

#9 Updated by okurz about 1 year ago

SLindoMansilla wrote:

okurz wrote:

SLindoMansilla wrote:

  • Contra: There is no clean way to detect in a test module if we are in alpha phase yet

I think there is a very clean way: if get_var('BETA') …

GMC phase?

well then invent another variable or read out the product string from /etc/os-release

#10 Updated by SLindoMansilla about 1 year ago

okurz wrote:

well then invent another variable or read out the product string from /etc/os-release

Yes, this is why I said that the discussion and decision from #76966 is important.
So, this ticket should wait for the other.

#11 Updated by SLindoMansilla 9 months ago

  • Assignee deleted (SLindoMansilla)

No time to work on this :(

#12 Updated by okurz 8 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: extra_tests_textmode_phub@s390x-kvm-sle15
https://openqa.suse.de/tests/5749805

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#13 Updated by okurz 8 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: extra_tests_textmode_phub@s390x-kvm-sle15
https://openqa.suse.de/tests/5814945

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

Also available in: Atom PDF