QA » openQA Project » openQA Tests

okurz wrote:

• is the runtime of individual scenarios the problems, e.g. the 50 minutes, or the time until jobs actually start?

Clarified this specific point with behlert: It's the runtime of individual scenarios that take "significantly longer than 1h".

Ideas we could follow on with:

• publish hdd image and run subsequent test modules in parallel jobs
• run sections of the longest scenario in multiple installation based scenarios, e.g. instead of "inst+a+b+c" and "inst2" do "inst+a" and "inst2+b+c"

with #46682 the select_patterns_and_packages step should also be much faster. https://openqa.suse.de/tests/2470033 is the latest job in the scenario sle-15-SP1-Installer-DVD-Staging:C-x86_64-minimal+base@64bit-staging which took 46m so some minutes saved.

The longest scenario in that build is the gnome one, e.g. see https://openqa.suse.de/tests/2470038 with 54m.

$ curl -s https://openqa.suse.de/tests/2470038/file/autoinst-log.txt | sed -n 's/^.*||| finished \([a-z_]*\).* (\([0-9]* s\))/\2 \1/p' | sort -n | tail -n 10
66 s yast
91 s reboot_gnome
93 s welcome
111 s bootloader
117 s addon_products_sle
129 s glxgears
253 s yast
264 s force_scheduled_tasks
278 s logs_from_installation_system
434 s await_install

We can not do anything about "await_install" but the other modules should be investigated.

and for "minimal+base"

$ curl -s https://openqa.suse.de/tests/2470033/file/autoinst-log.txt | sed -n 's/^.*||| finished \([a-z_]*\).* (\([0-9]* s\))/\2 \1/p' | sort -n | tail -n 10
84 s apache_nss
92 s welcome
97 s apache_ssl
105 s rsync
110 s bootloader
118 s addon_products_sle
142 s force_scheduled_tasks
211 s select_patterns_and_packages
242 s await_install
278 s logs_from_installation_system

Hm. I already told you in person that the run-time is the issue, not the waiting time. What question is open?
For completeness:
https://openqa.suse.de/tests/2492880 - 56:43 minutes
https://openqa.suse.de/tests/2492874 - 50:49 minutes
https://openqa.suse.de/tests/2492878 - 48:18 minutes
https://openqa.suse.de/tests/2492875 - 29:34 minutes
https://openqa.suse.de/tests/2492879 - 17:35 minutes
https://openqa.suse.de/tests/2492877 - 16:36 minutes
https://openqa.suse.de/tests/2492876 - 2:32 minutes

If we can keep all below 60 minutes, we are fine. Just a random example (Staging G.168.5)

@sbehlert thanks, that helps already.

The question that is still open:

okurz wrote:

• could you please add information which modules you don't consider important and why?

I will set up a meeting between Qa and RMs and discuss that topic.

• what modules are useful for RMs and QA?
• how long does it take to test them?
• document modules to make sure that testcases are stable

Reminder: staging testcases are expensive!

okurz wrote:

@sbehlert thanks, that helps already.

The question that is still open:

okurz wrote:

• could you please add information which modules you don't consider important and why?

I would expect that modules were added for a reason. Unfortunately there seems to be no history/changelog why something was added so there's a risk of removing somethign where a serious reason exists for its addition.

I would look at it from a different perspective:
The staging tests should fulfill these criteria/purpose

• ensure the images after acceptance are still installable
• prevent for some critical parts a complete breakage.

The RAID tests and encrypted partiton tests e.g. fulfill the first one.
A test if Firefox is still working, or sshd works would fall under point two for SLED resp. SLES.
Now, looking at the test scenarios, I wonder if e.g. these tests are really worth being checked in the stagings:

• mysql_srv
• rsync
• shibboleth
• vim

And, is it worth to have some modules in more than one scenarios in the stagings (for the "normal" testing, the answer is yes, but for staging I wonder):

• curl_https, sshd, zypper*

Note that all of these require a thorough look at, and can only be starting points. There might be others, and for sure it makes sense to check how much time each of the modules costs. It doesn't make sense to remove 5 testmodules that are done each in 5 seconds, and keep one that takes 60 minutes (exaggerating on purpose;) )

In an ideal world the staging testscenarios would also be dynamically adapted by the packages inside of a staging.
(Let's say, postgresql is added to a staging, then a "staging_test_database" scenario would be activated and run automatically by the test system).

But I admit, this is future technology ;) and goes way beyond the question

Hello Rodion, I have added you as watcher to this ticket. I learned from Rado today that you plan to add further tests to the Staging areas. I think we have to meet with the RMs and discuss that in more details. I plan to set up a meeting once I am back from China.

maritawerner wrote:

Hello Rodion, I have added you as watcher to this ticket. I learned from Rado today that you plan to add further tests to the Staging areas. I think we have to meet with the RMs and discuss that in more details. I plan to set up a meeting once I am back from China.

Hello Marita. Actually we have discussed only AutoYaST test for Y as of now, which is a good smoke test not to block AY testing for the candidate build. But in general we aim to improve earlier testing, so no plan to massively extend staging tests. As for general discussion, I still think it can be useful. E.g. I've learned from Rado that he would like to learn about missing packages on staging rather than in complete build and we can definitely improve there.

@marita @riafarov I think new scenarios, e.g. the mentioned autoyast scenario, is unrelated to the runtime of each individual scenario. sbehlert has provided a good answer and also we have reached the original goal again to bring test runtime below 1h. Based on the collected information we have created dependant tasks and will wait for these to be finished before we look at the whole picture again and see what else we can improve.

Thanks for your input here but since Stefan and I have agreed on having a more generic meeting of QA & RMs we will do that.

Update: we will have a meeting after SUSECon, so second week of April

See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html