action #113435
Updated by okurz almost 2 years ago
## Observation
openQA test in scenario sle-15-SP4-Server-DVD-Updates-x86_64-qam-regression-firefox-SLES@64bit fails in
[firefox_downloading](https://openqa.suse.de/tests/9099496/modules/firefox_downloading/steps/69)
due to empty download list expected but not empty. The individual entries are selected to be removed from the download list with the hotkey "d" pressed multiple times but apparently this does not work reliably. Interestingly in https://openqa.suse.de/tests/9099496#step/firefox_downloading/78 just after the post_fail_hook was triggered the download list is actually empty.
## Test suite description
Testsuite maintained at https://gitlab.suse.de/qa-maintenance/qam-openqa-yml.
## Reproducible
https://openqa.suse.de/tests/9099496#comments states that the issue is reproducible however it seems to still be sporadic. Judging from https://openqa.suse.de/tests/9099496#next_previous the issue has a fail ratio of about 50%.
## Impact
I found this test failure reviewing SLE maintenance test results which were blocking automatic approval meaning that test failures like these block automatic SLE maintenance update releases hence the issue should be treated with urgency.
## Expected result
Either the test should be fixed to have a reliably cleared download list with either hotkeys or mouse clicks or maybe that part of the test can be removed and is not necessary to maintain anymore. I understand how integration with the underlying OS should be tested, e.g. that an open file dialog works, but cleaning the download list is very much a Firefox-only test where we can rely on upstream behaviour not needing downstream tests.
## Suggestions
* Update the test description as it has no specific details for this test suite
* Update the test module maintainer, `wnereiz <wnereiz@github>` is outdated and should have a proper SUSE contact team/person
* Consider making the test more resilient as the test module seems to be rather unstable and has no stable design, e.g. the reliance on a hardcoded Leap ISO medium is a bad choice
* Verify stability with the approach outlined in https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation
## Rollback steps
Enable module again in test schedule, see https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15197
## Further details
Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=qam-regression-firefox-SLES&version=15-SP4)