QA » openQA Project » openQA Tests

SLindoMansilla wrote:

• Contra: There is no clean way to detect in a test module if we are in alpha phase yet

I think there is a very clean way: if get_var('BETA') …

The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.

• Software from supported SLE modules is not tested during alpha phase

can you clarify that

okurz wrote:

SLindoMansilla wrote:

• Contra: There is no clean way to detect in a test module if we are in alpha phase yet

I think there is a very clean way: if get_var('BETA') …

GMC phase?

The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.

• Software from supported SLE modules is not tested during alpha phase

can you clarify that

sssd test is not working because it fails trying to enable Package Hub.

SLindoMansilla wrote:

okurz wrote:

SLindoMansilla wrote:

• Contra: There is no clean way to detect in a test module if we are in alpha phase yet

I think there is a very clean way: if get_var('BETA') …

GMC phase?

The additional benefits of "make them soft-failing during alpha phase" are to not need to do any human action at end of beta phase to expect different behaviour, have the test code state the real expectations explicitly, be future proof for further versions and also ensuring that if package hub is still not available by mistake then the test should alarm about that. Otherwise I fear that either people "forget" to enable the test at the right time or ticket references are carried over for too long and ignored.

• Software from supported SLE modules is not tested during alpha phase

can you clarify that

sssd test is not working because it fails trying to enable Package Hub.

To clarify further: sssd tests depend on python-pam, and for instance, machinery is enabled (but that's a phub project), we might have other tests that do depend on phub aswell...

When is BETA variable supposed to be removed?

jlausuch wrote:

When is BETA variable supposed to be removed?

With GMC (Gold Master Candidate). But :), when we have the candidate build, we actually have a GMC Candidate, also GMCC, which means that is BETA yet, and therefore it still needs the beta handling (beta popup), after the GMCC is declared GMC, the beta popup disappears and we can remove the beta flag. But :), if we have two versions in the tests, e.g. 15-SP3, 12-SP6, it has to be set in YAML schedule per product, because they are not going to be GMC nor alpha at the same time.

SLindoMansilla wrote:

okurz wrote:

SLindoMansilla wrote:

• Contra: There is no clean way to detect in a test module if we are in alpha phase yet

I think there is a very clean way: if get_var('BETA') …

GMC phase?

well then invent another variable or read out the product string from /etc/os-release

okurz wrote:

well then invent another variable or read out the product string from /etc/os-release

Yes, this is why I said that the discussion and decision from #76966 is important.
So, this ticket should wait for the other.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: extra_tests_textmode_phub@s390x-kvm-sle15
https://openqa.suse.de/tests/5749805

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: extra_tests_textmode_phub@s390x-kvm-sle15
https://openqa.suse.de/tests/5814945

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed