coordination #76960
open[qe-core] Test modules that depends on PackageHub for software provided in supported SLE modules
0%
Description
Motivation¶
- Goal: Reduce human effort reviewing failed jobs
- Problem: PackageHub is not available until BETA1, which causes non-sporadic failing test modules
Proposals¶
keep them failing during Alpha phase¶
- Pros: Take over bot can label consequent failures without human intervention once a job is labeled
- Contra:
- Red color during alpha phase for known bug
- Software from supported SLE modules is not tested during alpha phase
make them soft-failing during alpha phase and failing since BETA1 phase if PHub is still not available/working¶
- Pros: Non-red (orange) color during alpha phase for known bug
- Contra:
- There is no clean way to detect in a test module if we are in alpha phase yet
- Software from supported SLE modules is not tested during alpha phase
change test modules to not depend on PackageHub¶
- Pros: Proper testing of supported software during alpha phase
- Contra: Implementation requires big human effort
Updated by okurz over 3 years ago
SLindoMansilla wrote:
- Contra: There is no clean way to detect in a test module if we are in alpha phase yet
I think there is a very clean way: if get_var('BETA') …
The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.
- Software from supported SLE modules is not tested during alpha phase
can you clarify that
Updated by SLindoMansilla over 3 years ago
okurz wrote:
SLindoMansilla wrote:
- Contra: There is no clean way to detect in a test module if we are in alpha phase yet
I think there is a very clean way:
if get_var('BETA') …
GMC phase?
The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.
- Software from supported SLE modules is not tested during alpha phase
can you clarify that
sssd test is not working because it fails trying to enable Package Hub.
Updated by szarate over 3 years ago
SLindoMansilla wrote:
okurz wrote:
SLindoMansilla wrote:
- Contra: There is no clean way to detect in a test module if we are in alpha phase yet
I think there is a very clean way:
if get_var('BETA') …
GMC phase?
The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.
- Software from supported SLE modules is not tested during alpha phase
can you clarify that
sssd test is not working because it fails trying to enable Package Hub.
To clarify further: sssd tests depend on python-pam, and for instance, machinery is enabled (but that's a phub project), we might have other tests that do depend on phub aswell...
Updated by jlausuch over 3 years ago
When is BETA variable supposed to be removed?
Updated by SLindoMansilla over 3 years ago
jlausuch wrote:
When is BETA variable supposed to be removed?
With GMC (Gold Master Candidate). But :), when we have the candidate build, we actually have a GMC Candidate, also GMCC, which means that is BETA yet, and therefore it still needs the beta handling (beta popup), after the GMCC is declared GMC, the beta popup disappears and we can remove the beta flag. But :), if we have two versions in the tests, e.g. 15-SP3, 12-SP6, it has to be set in YAML schedule per product, because they are not going to be GMC nor alpha at the same time.
Updated by okurz over 3 years ago
SLindoMansilla wrote:
okurz wrote:
SLindoMansilla wrote:
- Contra: There is no clean way to detect in a test module if we are in alpha phase yet
I think there is a very clean way:
if get_var('BETA') …
GMC phase?
well then invent another variable or read out the product string from /etc/os-release
Updated by SLindoMansilla over 3 years ago
okurz wrote:
well then invent another variable or read out the product string from /etc/os-release
Yes, this is why I said that the discussion and decision from #76966 is important.
So, this ticket should wait for the other.
Updated by SLindoMansilla about 3 years ago
- Assignee deleted (
SLindoMansilla)
No time to work on this :(
Updated by okurz about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: extra_tests_textmode_phub@s390x-kvm-sle15
https://openqa.suse.de/tests/5749805
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by okurz about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: extra_tests_textmode_phub@s390x-kvm-sle15
https://openqa.suse.de/tests/5814945
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by szarate almost 2 years ago
- Tags deleted (
functional, Yast, migration, qac) - Subject changed from Test modules that depends on PackageHub for software provided in supported SLE modules to [qe-core] Test modules that depends on PackageHub for software provided in supported SLE modules
Updated by slo-gin about 1 month ago
This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.