action #36100
closed[functional][u] Better explicit system performance test
0%
Description
Motivation¶
Our openQA tests are very good at finding system performance issues, e.g. bsc#1063638 and bsc#1032831, but very bad at revealing the actual problems to test reviewers because seemingly random test modules fail sporadically due to just timeouts, windows not popping up, etc. . Recording random soft-fails in random scenarios is apparently not the way. We should define specific scenarios to test everything we do not want to test in other scenarios. Whenever we encounter other tests failing we should ensure that they work in a stable way and corresponding test steps are moved into the new "system performance" test scenario
Acceptance criteria¶
- AC1: bsc#1063638 is not tracked in any other test scenario than one for system performance testing
Tasks¶
- Track in development for TW
- Add to Leap
- Add to SLE15
- Add to SLE12SP4
- Look for more stuff to add in this scenario, e.g. something about yast2_snapper which was recently removed by lnussel from standard scenarios, "reboot and call krunner test", see
Updated by okurz over 6 years ago
- Description updated (diff)
Added new test suite system_performance on o3 with settings
DESKTOP=gnome
SOFTFAIL_BSC1063638=1
START_AFTER_TEST=gnome
as well as update_Leap_42.3_kde+system_performance.
Added both to Development Tumbleweed
A new TW snapshot triggered
Updated by okurz over 6 years ago
hm, both tests passed. They should have been soft-failed. But at least the latest job in this scenario https://openqa.opensuse.org/tests/681318 showed the soft-fail, that's ok.
Updated by okurz over 6 years ago
Added scenarios to the validation job groups of Leap and TW including aarch64 and ppc64le, to be monitored: https://openqa.opensuse.org/tests?match=system_performance
Updated by okurz over 6 years ago
- Due date changed from 2018-06-05 to 2018-06-19
- Target version changed from Milestone 16 to Milestone 17
aarch64 and ppc64le ones are fine. Next to check: SLE
Updated by okurz over 6 years ago
- Priority changed from Normal to Low
Added one test suite "system_performance" on osd and scheduled in "test development" group of SLE12SP4 and SLE15, let's see at the end of the sprint the results.
Check https://openqa.suse.de/tests?match=system_performance for latest results.
Updated by okurz over 6 years ago
- Status changed from Feedback to Resolved
Tests on SLE12SP4 looked fine, no new build for SLE15 expected.
- x86_64
- aarch64 failed in same product bug as other scenarios in same build
- s390x failed trying to switch to a display manager but that seems to be something we need to handle anyway on a bigger scale
- ppc64le is fine as well
Moved tests to functional job groups of SLE15 (as template for following versions) and SLE12SP4
Updated by okurz over 6 years ago
- Target version changed from Milestone 17 to Milestone 17
Updated by okurz about 6 years ago
- Related to action #20036: [desktop]x11_setup needs to run after force_cron_runs added
Updated by okurz about 6 years ago
- Related to action #39059: [sle][functional][y] detect "openSUSE sucks bug" about btrfs balance and record_soft_fail (was: yast2_gui tests modules as application could not start up) added