QA » openQA Project » openQA Infrastructure

Presented the current findings in the weekly QE Sync. People supported that only SLE maintenance tests are impacted and are likely also the source of problem.

To improve we identified the following areas:

• Network: We are mainly running on 1G network so communication between central OSD instance and workers is sub-optimal as well as connection outside the OSD network
• Storage: Some of our storage is slow and hence tests are slow to start, see #96554
• more workers: There is still some capacity for more workers (although we should be careful to not overload the central OSD instance)
• test design efficiency: Many tests use the interactive GUI installer and take long to add all repos, re-download the same set of updates all the time instead of relying on something like nightly snapshots, etc.

ideas:

• cancel older scheduled investigation jobs to crosscheck if they really don't have a significant impact. did that now with an SQL query

openqa=> delete from jobs where state='scheduled' and test ~ ':investigate:' and t_created <= '2021-08-16';
DELETE 2643

Checking 3 hours later I see that the job schedule seems to still increase again. I checked a random maintenance incident test overview, e.g. https://openqa.suse.de/tests/overview?build=:20659:java-1_8_0-openjdk&distri=sle&version=15-SP1&groupid=236 and saw 26 jobs. but

openqa=> select count(id) from jobs where state='scheduled' and build=':20659:java-1_8_0-openjdk';
 count 
-------
    79

so multiple sets of the same are scheduled. That's from the back of the queue. https://openqa.suse.de/tests/overview?build=:19994:ffmpeg&distri=sle&version=15&groupid=176 shows 130 jobs.

The according SQL query shows 320 jobs so again multiple sets.

• create specific tickets about the above improvement points. Otherwise I assume no squad would pick that up anyway

okurz:

Checking 3 hours later I see that the job schedule seems to still increase again. I checked a random maintenance incident test overview, e.g. https://openqa.suse.de/tests/overview?build=:20659:java-1_8_0-openjdk&distri=sle&version=15-SP1&groupid=236 and saw 26 jobs. but

openqa=> select count(id) from jobs where state='scheduled' and build=':20659:java-1_8_0-openjdk';
 count 
-------
    79

so multiple sets of the same are scheduled. That's from the back of the queue. https://openqa.suse.de/tests/overview?build=:19994:ffmpeg&distri=sle&version=15&groupid=176 shows 130 jobs.

not same, different flavors, every incident has multiple product versions and usually multiple products + if is change in incident repo --> new shedule

so for example incident 19994 ffmpeg: https://smelt.suse.de/incident/19994/ and look at 'Repositories' table (and this incident is on smaller size)

yes, you are right, stupid me. https://openqa.suse.de/tests/overview?build=:20659:java-1_8_0-openjdk shows the correct number.

Any different idea why we seem to have that many jobs recently?

okurz wrote:

yes, you are right, stupid me. https://openqa.suse.de/tests/overview?build=:20659:java-1_8_0-openjdk shows the correct number.

Any different idea why we seem to have that many jobs recently

we have a pretty high amount of incidents in testing QUEUE and even more surprisingly in the staging area. We can lower the amount of tested incidents if we stop tests incidents in staging area ( cons: are -> Incident tests in openQA will start the only when in moved to testing ( so on some incidents few hours added delay) and we will lose early warning if is incident broken in openQA tests)