Project

General

Profile

action #36100

[functional][u] Better explicit system performance test

Added by okurz over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Enhancement to existing tests
Target version:
SUSE QA - Milestone 17
Start date:
2018-05-11
Due date:
2018-06-19
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

Our openQA tests are very good at finding system performance issues, e.g. bsc#1063638 and bsc#1032831, but very bad at revealing the actual problems to test reviewers because seemingly random test modules fail sporadically due to just timeouts, windows not popping up, etc. . Recording random soft-fails in random scenarios is apparently not the way. We should define specific scenarios to test everything we do not want to test in other scenarios. Whenever we encounter other tests failing we should ensure that they work in a stable way and corresponding test steps are moved into the new "system performance" test scenario

Acceptance criteria

  • AC1: bsc#1063638 is not tracked in any other test scenario than one for system performance testing

Tasks

  • Track in development for TW
  • Add to Leap
  • Add to SLE15
  • Add to SLE12SP4
  • Look for more stuff to add in this scenario, e.g. something about yast2_snapper which was recently removed by lnussel from standard scenarios, "reboot and call krunner test", see

Related issues

Related to openQA Tests - action #20036: [desktop]x11_setup needs to run after force_cron_runsRejected2017-06-26

Related to openQA Tests - action #39059: [sle][functional][y] detect "openSUSE sucks bug" about btrfs balance and record_soft_fail (was: yast2_gui tests modules as application could not start up)Resolved2018-08-012018-10-09

History

#1 Updated by okurz over 4 years ago

  • Description updated (diff)

Added new test suite system_performance on o3 with settings

DESKTOP=gnome
SOFTFAIL_BSC1063638=1
START_AFTER_TEST=gnome

as well as update_Leap_42.3_kde+system_performance.

Added both to Development Tumbleweed

A new TW snapshot triggered

#2 Updated by okurz over 4 years ago

hm, both tests passed. They should have been soft-failed. But at least the latest job in this scenario https://openqa.opensuse.org/tests/681318 showed the soft-fail, that's ok.

#3 Updated by okurz over 4 years ago

Added scenarios to the validation job groups of Leap and TW including aarch64 and ppc64le, to be monitored: https://openqa.opensuse.org/tests?match=system_performance

#4 Updated by okurz over 4 years ago

  • Due date changed from 2018-06-05 to 2018-06-19
  • Target version changed from Milestone 16 to Milestone 17

aarch64 and ppc64le ones are fine. Next to check: SLE

#5 Updated by okurz over 4 years ago

  • Priority changed from Normal to Low

Added one test suite "system_performance" on osd and scheduled in "test development" group of SLE12SP4 and SLE15, let's see at the end of the sprint the results.

Check https://openqa.suse.de/tests?match=system_performance for latest results.

#6 Updated by okurz over 4 years ago

  • Status changed from In Progress to Feedback

#7 Updated by okurz over 4 years ago

  • Status changed from Feedback to Resolved

Tests on SLE12SP4 looked fine, no new build for SLE15 expected.

  • x86_64
  • aarch64 failed in same product bug as other scenarios in same build
  • s390x failed trying to switch to a display manager but that seems to be something we need to handle anyway on a bigger scale
  • ppc64le is fine as well

Moved tests to functional job groups of SLE15 (as template for following versions) and SLE12SP4

#8 Updated by okurz over 4 years ago

  • Target version changed from Milestone 17 to Milestone 17

#9 Updated by okurz about 4 years ago

  • Related to action #20036: [desktop]x11_setup needs to run after force_cron_runs added

#10 Updated by okurz about 4 years ago

  • Related to action #39059: [sle][functional][y] detect "openSUSE sucks bug" about btrfs balance and record_soft_fail (was: yast2_gui tests modules as application could not start up) added

Also available in: Atom PDF