Project

General

Profile

Actions

action #36100

closed

[functional][u] Better explicit system performance test

Added by okurz almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Enhancement to existing tests
Target version:
SUSE QA - Milestone 17
Start date:
2018-05-11
Due date:
2018-06-19
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

Our openQA tests are very good at finding system performance issues, e.g. bsc#1063638 and bsc#1032831, but very bad at revealing the actual problems to test reviewers because seemingly random test modules fail sporadically due to just timeouts, windows not popping up, etc. . Recording random soft-fails in random scenarios is apparently not the way. We should define specific scenarios to test everything we do not want to test in other scenarios. Whenever we encounter other tests failing we should ensure that they work in a stable way and corresponding test steps are moved into the new "system performance" test scenario

Acceptance criteria

  • AC1: bsc#1063638 is not tracked in any other test scenario than one for system performance testing

Tasks

  • Track in development for TW
  • Add to Leap
  • Add to SLE15
  • Add to SLE12SP4
  • Look for more stuff to add in this scenario, e.g. something about yast2_snapper which was recently removed by lnussel from standard scenarios, "reboot and call krunner test", see

Related issues 2 (0 open2 closed)

Related to openQA Tests - action #20036: [desktop]x11_setup needs to run after force_cron_runsRejectedGraceWang2017-06-26

Actions
Related to openQA Tests - action #39059: [sle][functional][y] detect "openSUSE sucks bug" about btrfs balance and record_soft_fail (was: yast2_gui tests modules as application could not start up)Resolvedriafarov2018-08-012018-10-09

Actions
Actions #1

Updated by okurz almost 6 years ago

  • Description updated (diff)

Added new test suite system_performance on o3 with settings

DESKTOP=gnome
SOFTFAIL_BSC1063638=1
START_AFTER_TEST=gnome

as well as update_Leap_42.3_kde+system_performance.

Added both to Development Tumbleweed

A new TW snapshot triggered

Actions #2

Updated by okurz almost 6 years ago

hm, both tests passed. They should have been soft-failed. But at least the latest job in this scenario https://openqa.opensuse.org/tests/681318 showed the soft-fail, that's ok.

Actions #3

Updated by okurz almost 6 years ago

Added scenarios to the validation job groups of Leap and TW including aarch64 and ppc64le, to be monitored: https://openqa.opensuse.org/tests?match=system_performance

Actions #4

Updated by okurz over 5 years ago

  • Due date changed from 2018-06-05 to 2018-06-19
  • Target version changed from Milestone 16 to Milestone 17

aarch64 and ppc64le ones are fine. Next to check: SLE

Actions #5

Updated by okurz over 5 years ago

  • Priority changed from Normal to Low

Added one test suite "system_performance" on osd and scheduled in "test development" group of SLE12SP4 and SLE15, let's see at the end of the sprint the results.

Check https://openqa.suse.de/tests?match=system_performance for latest results.

Actions #6

Updated by okurz over 5 years ago

  • Status changed from In Progress to Feedback
Actions #7

Updated by okurz over 5 years ago

  • Status changed from Feedback to Resolved

Tests on SLE12SP4 looked fine, no new build for SLE15 expected.

  • x86_64
  • aarch64 failed in same product bug as other scenarios in same build
  • s390x failed trying to switch to a display manager but that seems to be something we need to handle anyway on a bigger scale
  • ppc64le is fine as well

Moved tests to functional job groups of SLE15 (as template for following versions) and SLE12SP4

Actions #8

Updated by okurz over 5 years ago

  • Target version changed from Milestone 17 to Milestone 17
Actions #9

Updated by okurz over 5 years ago

  • Related to action #20036: [desktop]x11_setup needs to run after force_cron_runs added
Actions #10

Updated by okurz over 5 years ago

  • Related to action #39059: [sle][functional][y] detect "openSUSE sucks bug" about btrfs balance and record_soft_fail (was: yast2_gui tests modules as application could not start up) added
Actions

Also available in: Atom PDF