QA (public)

Notes:

• By definition the tests listed here are no longer there
• It might be useful to lookup previous jobs for these cases
• A diff of the YAML (if the YAML was changed) and GitLab diff if applicable

Suggested approach:

• Include descriptions of tests that previously passed but were removed

vpelcak wrote:

Acceptance criteria¶

• AC1: it is clear to the tester what he needs to do when he sees message like the one above

This sounds like extending the message "Please, check openQA manually". You would only expect better instructions within that text, right? Or elsewhere as well?

• AC2: ideally, information about what testing needs to be done to cover for the disabled testcase should be easily accessible

That implies that manual testing would be able to do what an automated openQA test scenario based on a complete test suite does. It would be possible for removed test modules, e.g. just the "firefox" test module, but AFAIK that is not handled at all by test report templates. But I see that as effectively impossible when complete test scenarios are unscheduled. I know that this comes up from time to time that people think they can do manually for single updates what an automated test in openQA can do but only a very tiny subset of tests would be possible to execute manually nowadays within a reasonable time.

I am just picking one of the above mentioned test suites, possibly the smallest one, "mau-extratests-zypper". Take for example https://openqa.suse.de/tests/6224114 showing one successful run of such scenario. The test is composed of the test modules. Let's imagine we could ask a human tester to follow all instructions from the test modules. That is effectively following all instructions that show up also in the log file https://openqa.suse.de/tests/6224114/file/autoinst-log.txt . One could search for all "<<<" markers which are commands that openQA types. But that accounts for 2300 lines in the mentioned example, unlikely to be followed by humans. Instead let's look at modules themselves. First "boot_to_desktop", ok, feasible to achieve by a tester just judging from the name. Then comes "zypper_lr_validate". In openQA this takes 8s. The instructions are visible in https://openqa.suse.de/tests/6224114/modules/zypper_lr_validate/steps/1/src corresponding to https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/console/zypper_lr_validate.pm . Very hard to read as the test module needs to be very special depending on SLE service packs. But a test for an incident update likely needs to understand these specifics as well. And then there are many more test modules, like "zypper_ref", "zypper_info", "check_interactive_flag", "validate_packages_and_patterns", "zypper_extend", "coredump_collect". Again, asking humans to redo what an automated test system does is impossible.

The best approach I see is:

1. If tests produce false positives and it wasn't a mistake that the tests have been added, e.g. someone just added new tests that were never there, then revert what introduces the test regression and retrigger tests for the incident updates
2. If in the rare exception that one really really really thinks it's the right approach to revert to manual testing, e.g. urgent security fix while all openQA tests are broken due to infrastructure problem, then rely on expertise of professional testers of how the update can be tested best as we can not formalize consistent and complete manual test instructions that could replace the automated tests

I gave this high priority, because there is a danger, that additional testing won't happen and updates are released with reduced test coverage with risk of regressions.

ok, but it's not a recent regression, is it? So basically as a safeguard or workaround until we find a better solution we/you should remind or explain manually what it means.

okurz wrote:

vpelcak wrote:

Acceptance criteria¶

• AC1: it is clear to the tester what he needs to do when he sees message like the one above

This sounds like extending the message "Please, check openQA manually". You would only expect better instructions within that text, right? Or elsewhere as well?

Yes. Having better instructions what needs to be done in general would help. The block of the text below, while great start, is hard to decode for the people.

• AC2: ideally, information about what testing needs to be done to cover for the disabled testcase should be easily accessible

That implies that manual testing would be able to do what an automated openQA test scenario based on a complete test suite does. It would be possible for removed test modules, e.g. just the "firefox" test module, but AFAIK that is not handled at all by test report templates. But I see that as effectively impossible when complete test scenarios are unscheduled. I know that this comes up from time to time that people think they can do manually for single updates what an automated test in openQA can do but only a very tiny subset of tests would be possible to execute manually nowadays within a reasonable time.

I see your point. OTOH, we used to work in that kind of mode already. And I certainly hope that this situation won't be too frequent.

The best approach I see is:

1. If tests produce false positives and it wasn't a mistake that the tests have been added, e.g. someone just added new tests that were never there, then revert what introduces the test regression and retrigger tests for the incident updates
2. If in the rare exception that one really really really thinks it's the right approach to revert to manual testing, e.g. urgent security fix while all openQA tests are broken due to infrastructure problem, then rely on expertise of professional testers of how the update can be tested best as we can not formalize consistent and complete manual test instructions that could replace the automated tests

You listed just few of possible options. There could be a version bump causing testcase not working, long time unstable test...

I gave this high priority, because there is a danger, that additional testing won't happen and updates are released with reduced test coverage with risk of regressions.

ok, but it's not a recent regression, is it? So basically as a safeguard or workaround until we find a better solution we/you should remind or explain manually what it means.

No, it is not a regression indeed.

The best approach I see is:

1. If tests produce false positives and it wasn't a mistake that the tests have been added, e.g. someone just added new tests that were never there, then revert what introduces the test regression and retrigger tests for the incident updates
2. If in the rare exception that one really really really thinks it's the right approach to revert to manual testing, e.g. urgent security fix while all openQA tests are broken due to infrastructure problem, then rely on expertise of professional testers of how the update can be tested best as we can not formalize consistent and complete manual test instructions that could replace the automated tests

You listed just few of possible options. There could be a version bump causing testcase not working, long time unstable test...

That makes no difference to me as still a human tester will not be possible to conduct the test in depth as the automated tests are able to do. What I have not mentioned as option is of course what should have been decided in before: That such version update can only be approved as soon as the automated tests have been updated with exception of what I described as option 2 which should not apply for a "version bump causing testcase not working". The risk of releasing a product regression breaking customer use cases is just too great.

As the list of missing tests can get long I would suggest to abbreviate after e.g. 3 lines with … and point to a test overview page on openQA instead, e.g. https://openqa.suse.de/tests/overview?distri=sle&build=:19924:webkit2gtk3
but as we can not store openQA jobs indefinitely this has a limitation. All openQA results are deleted unless marked as "important". Rather than "saving the data elsewhere because we don't have enough space in openQA" I would say if we need results for longer in openQA then we will find a solution in openQA. So as alternative to saving a list of tests elsewhere we can discuss keeping results longer for certain job groups or we can mark builds or jobs as "important" and then they are kept around longer. This needs corresponding space on OSD reserved for that purpose of course.

Hello.
How is the situation about this?
I feel that the priority is way too low.

When I said that it is not a regression, I think I was not correct.

Yes, it is not a regression in the matter of notifications to the people, but every unscheduled test poses a risk of regression in the test coverage and subsequently in the product.
That needs to be addressed.

OTOH some unschedules could be for a good reason (useless test for example). Perhaps it is possible to somehow address that, too.

Maybe we should refine the ticket.

This issue is addressed by https://gitlab.suse.de/qa-maintenance/teregen/-/merge_requests/16 as explained inside #108812