Project

General

Profile

Actions

coordination #47192

closed

[sle][functional][y][epic] some openQA tests in staging take more than 50 minutes

Added by maritawerner almost 6 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Target version:
SUSE QA (private) - Milestone 25
Start date:
2019-02-06
Due date:
2019-06-04
% Done:

0%

Estimated time:
Difficulty:

Description

In the SLE Staging area https://build.suse.de/project/staging_projects/SUSE:SLE-15-SP1:GA
a lot of testcases take more than 50 Minutes.
The RMs have complained that that is to long and asked us to review the Testcases and skip less important test modules, e.g. shibboleth
Example: https://openqa.suse.de/tests/2436298 in Staging C

Further details

Link to


Related issues 3 (0 open3 closed)

Related to openQA Tests (public) - action #46682: [functional][y] lvm test need more than 2 hours to run - optimize select_patterns_and_packagesResolvedJERiveraMoya2019-01-252019-02-26

Actions
Blocked by openQA Tests (public) - action #47102: [functional][y] soft-fail bsc#1092088 in logs_from_installation_system - but maybe not in *all* installation scenarios?Resolvedriafarov2018-12-212019-02-26

Actions
Blocked by openQA Tests (public) - action #48035: [functional][y] execute SUT setup steps only once per test scenarioRejectedriafarov2019-02-062019-04-23

Actions
Actions #1

Updated by okurz almost 6 years ago

  • Related to action #46682: [functional][y] lvm test need more than 2 hours to run - optimize select_patterns_and_packages added
Actions #2

Updated by okurz almost 6 years ago

  • Subject changed from [sle][functional] some openQA tests in Staging take more than 50 minutes to [sle][functional][y][u] some openQA tests in staging take more than 50 minutes
  • Category set to Enhancement to existing tests
  • Status changed from New to Feedback
  • Target version set to Milestone 23

Thanks for the report maritawerner.

Not as an "excuse" but to find out if this is a regression I looked for an older job and found https://openqa.suse.de/tests/2283578# as the oldest. This one also took 56m so equivalent.

A part of it we already try to adress in #46682

#46682#note-4 has an investigation how we evaluate the time impact

@sbehlert

  • could you please add information which modules you don't consider important and why?
  • is the runtime of individual scenarios the problems, e.g. the 50 minutes, or the time until jobs actually start?
Actions #3

Updated by okurz almost 6 years ago

  • Copied to action #47198: [qam][yellow] sshd test modules takes too long - negative impact on staging tests added
Actions #4

Updated by okurz almost 6 years ago

okurz wrote:

  • is the runtime of individual scenarios the problems, e.g. the 50 minutes, or the time until jobs actually start?

Clarified this specific point with behlert: It's the runtime of individual scenarios that take "significantly longer than 1h".

Ideas we could follow on with:

  • publish hdd image and run subsequent test modules in parallel jobs
  • run sections of the longest scenario in multiple installation based scenarios, e.g. instead of "inst+a+b+c" and "inst2" do "inst+a" and "inst2+b+c"
Actions #5

Updated by okurz almost 6 years ago

with #46682 the select_patterns_and_packages step should also be much faster. https://openqa.suse.de/tests/2470033 is the latest job in the scenario sle-15-SP1-Installer-DVD-Staging:C-x86_64-minimal+base@64bit-staging which took 46m so some minutes saved.

The longest scenario in that build is the gnome one, e.g. see https://openqa.suse.de/tests/2470038 with 54m.

$ curl -s https://openqa.suse.de/tests/2470038/file/autoinst-log.txt | sed -n 's/^.*||| finished \([a-z_]*\).* (\([0-9]* s\))/\2 \1/p' | sort -n | tail -n 10
66 s yast
91 s reboot_gnome
93 s welcome
111 s bootloader
117 s addon_products_sle
129 s glxgears
253 s yast
264 s force_scheduled_tasks
278 s logs_from_installation_system
434 s await_install

We can not do anything about "await_install" but the other modules should be investigated.

and for "minimal+base"

$ curl -s https://openqa.suse.de/tests/2470033/file/autoinst-log.txt | sed -n 's/^.*||| finished \([a-z_]*\).* (\([0-9]* s\))/\2 \1/p' | sort -n | tail -n 10
84 s apache_nss
92 s welcome
97 s apache_ssl
105 s rsync
110 s bootloader
118 s addon_products_sle
142 s force_scheduled_tasks
211 s select_patterns_and_packages
242 s await_install
278 s logs_from_installation_system
Actions #6

Updated by okurz almost 6 years ago

  • Copied to action #48035: [functional][y] execute SUT setup steps only once per test scenario added
Actions #7

Updated by okurz almost 6 years ago

  • Blocked by action #47102: [functional][y] soft-fail bsc#1092088 in logs_from_installation_system - but maybe not in *all* installation scenarios? added
Actions #8

Updated by okurz almost 6 years ago

  • Subject changed from [sle][functional][y][u] some openQA tests in staging take more than 50 minutes to [sle][functional][y][u][epic] some openQA tests in staging take more than 50 minutes
  • Description updated (diff)
  • Due date set to 2019-03-26
Actions #9

Updated by okurz almost 6 years ago

  • Copied to deleted (action #47198: [qam][yellow] sshd test modules takes too long - negative impact on staging tests)
Actions #10

Updated by okurz almost 6 years ago

  • Related to action #47198: [qam][yellow] sshd test modules takes too long - negative impact on staging tests added
Actions #11

Updated by okurz almost 6 years ago

  • Related to deleted (action #47198: [qam][yellow] sshd test modules takes too long - negative impact on staging tests)
Actions #12

Updated by okurz almost 6 years ago

  • Blocked by action #47198: [qam][yellow] sshd test modules takes too long - negative impact on staging tests added
Actions #13

Updated by okurz almost 6 years ago

  • Copied to deleted (action #48035: [functional][y] execute SUT setup steps only once per test scenario)
Actions #14

Updated by okurz almost 6 years ago

  • Blocked by action #48035: [functional][y] execute SUT setup steps only once per test scenario added
Actions #15

Updated by okurz almost 6 years ago

  • Due date changed from 2019-03-26 to 2019-06-04
  • Status changed from Feedback to Blocked
  • Priority changed from High to Normal
  • Target version changed from Milestone 23 to Milestone 25

no response from behlert (also pinged him in person and over IRC). Seems like we can reduce priority.

Blocked by two linked tasks now.

Actions #16

Updated by sbehlert almost 6 years ago

Hm. I already told you in person that the run-time is the issue, not the waiting time. What question is open?
For completeness:
https://openqa.suse.de/tests/2492880 - 56:43 minutes
https://openqa.suse.de/tests/2492874 - 50:49 minutes
https://openqa.suse.de/tests/2492878 - 48:18 minutes
https://openqa.suse.de/tests/2492875 - 29:34 minutes
https://openqa.suse.de/tests/2492879 - 17:35 minutes
https://openqa.suse.de/tests/2492877 - 16:36 minutes
https://openqa.suse.de/tests/2492876 - 2:32 minutes

If we can keep all below 60 minutes, we are fine. Just a random example (Staging G.168.5)

Actions #17

Updated by okurz almost 6 years ago

@sbehlert thanks, that helps already.

The question that is still open:

okurz wrote:

  • could you please add information which modules you don't consider important and why?
Actions #18

Updated by maritawerner almost 6 years ago

I will set up a meeting between Qa and RMs and discuss that topic.

  • what modules are useful for RMs and QA?
  • how long does it take to test them?
  • document modules to make sure that testcases are stable

Reminder: staging testcases are expensive!

Actions #19

Updated by sbehlert almost 6 years ago

okurz wrote:

@sbehlert thanks, that helps already.

The question that is still open:

okurz wrote:

  • could you please add information which modules you don't consider important and why?

I would expect that modules were added for a reason. Unfortunately there seems to be no history/changelog why something was added so there's a risk of removing somethign where a serious reason exists for its addition.

I would look at it from a different perspective:
The staging tests should fulfill these criteria/purpose

  • ensure the images after acceptance are still installable
  • prevent for some critical parts a complete breakage.

The RAID tests and encrypted partiton tests e.g. fulfill the first one.
A test if Firefox is still working, or sshd works would fall under point two for SLED resp. SLES.
Now, looking at the test scenarios, I wonder if e.g. these tests are really worth being checked in the stagings:

  • mysql_srv
  • rsync
  • shibboleth
  • vim

And, is it worth to have some modules in more than one scenarios in the stagings (for the "normal" testing, the answer is yes, but for staging I wonder):

  • curl_https, sshd, zypper*

Note that all of these require a thorough look at, and can only be starting points. There might be others, and for sure it makes sense to check how much time each of the modules costs. It doesn't make sense to remove 5 testmodules that are done each in 5 seconds, and keep one that takes 60 minutes (exaggerating on purpose;) )

In an ideal world the staging testscenarios would also be dynamically adapted by the packages inside of a staging.
(Let's say, postgresql is added to a staging, then a "staging_test_database" scenario would be activated and run automatically by the test system).

But I admit, this is future technology ;) and goes way beyond the question

Actions #20

Updated by maritawerner almost 6 years ago

Hello Rodion, I have added you as watcher to this ticket. I learned from Rado today that you plan to add further tests to the Staging areas. I think we have to meet with the RMs and discuss that in more details. I plan to set up a meeting once I am back from China.

Actions #21

Updated by riafarov almost 6 years ago

maritawerner wrote:

Hello Rodion, I have added you as watcher to this ticket. I learned from Rado today that you plan to add further tests to the Staging areas. I think we have to meet with the RMs and discuss that in more details. I plan to set up a meeting once I am back from China.

Hello Marita. Actually we have discussed only AutoYaST test for Y as of now, which is a good smoke test not to block AY testing for the candidate build. But in general we aim to improve earlier testing, so no plan to massively extend staging tests. As for general discussion, I still think it can be useful. E.g. I've learned from Rado that he would like to learn about missing packages on staging rather than in complete build and we can definitely improve there.

Actions #22

Updated by okurz almost 6 years ago

@marita @riafarov I think new scenarios, e.g. the mentioned autoyast scenario, is unrelated to the runtime of each individual scenario. sbehlert has provided a good answer and also we have reached the original goal again to bring test runtime below 1h. Based on the collected information we have created dependant tasks and will wait for these to be finished before we look at the whole picture again and see what else we can improve.

Actions #23

Updated by maritawerner over 5 years ago

Thanks for your input here but since Stefan and I have agreed on having a more generic meeting of QA & RMs we will do that.

Actions #24

Updated by maritawerner over 5 years ago

Update: we will have a meeting after SUSECon, so second week of April

Actions #25

Updated by okurz over 5 years ago

  • Assignee changed from okurz to riafarov

Move to new QSF-y PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-y. Feel free to reassign to me or someone else in this case. Thanks.

Actions #26

Updated by riafarov over 5 years ago

  • Blocked by deleted (action #47198: [qam][yellow] sshd test modules takes too long - negative impact on staging tests)
Actions #27

Updated by riafarov over 5 years ago

  • Status changed from Blocked to Resolved

As of now none of the test suites take more than 50 minutes under normal conditions. Please, feel free to reopen in case you want to reduce coverage even further.

Actions #28

Updated by SLindoMansilla over 5 years ago

  • Subject changed from [sle][functional][y][u][epic] some openQA tests in staging take more than 50 minutes to [sle][functional][y][epic] some openQA tests in staging take more than 50 minutes

Removing from U-Team

Actions #29

Updated by szarate about 4 years ago

  • Tracker changed from action to coordination
Actions

Also available in: Atom PDF