Project

General

Profile

Actions
Copy link

action #90857

closed

Add redundancy for SAP multi machines tests - Extend openQA worker config to accomodate for upgraded RAM

Added by jadamek over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Low
Assignee:
okurz
Category:
-
Target version:
openQA Project (public) - Ready
Start date:
Due date:
2021-08-03
% Done:

0%

Estimated time:

Description

OpenQA QEM review reported an issue with our SAP HANA tests executed on Maintenance TestRepo.
The need is to get more resources or executing the tests on more existing machines.

First, let me summarize the situation:
Nowadays, the timing is really tight because as you know, the maintenance test repo is triggered twice a day.
That means 2 X 6 OS versions to test (12-SP3 to 15-SP2) with one HANA test per OS version.
And it must be completed before the next build otherwise jobs are tagged obsolete.
One HANA test requires 49 GB RAM: 2 x 24 GB (HANA machines) + 1 GB for the support server machine.

For these tests, we are only using openqaworker8 (sap_sle12) and openqaworker9 (sap_sle15), we made it like that to preserve the memory usage of the openQA instance (https://progress.opensuse.org/issues/73246):
Like that the HANA tests are done in serialize for sle12 as well as sle15.

For instance:
HANA test starts for 15 GA on openqaworker9, the test lasts half an hour and a half. Once the test is done, the HANA test on 15 SP1 starts, and so on...
Like we have 3 differents 15 versions (GA, SP1, SP2), the tests last 4 hours and a half only for SLE15.
For SLE12, the HANA test lasts one hour so as we have 3 different 12 versions (SP3, SP4, SP5), the tests last 3 hours for SLE12. 12-S2P2 was removed recently.

Besides that, both workers are also used on Maintenance incident and we can not know how much we need there in advance.

I agree the solution isn't redundant at all. If one of the workers is down, the tests can not be executed elsewhere.
For speeding up the tests, we can think about adding memory in both workers (at least 64GB per worker, not less because the jobs are linked together as they are multi machines jobs).

Related issues 2 (0 open — 2 closed)

Related to openQA Infrastructure (public) - action #73246: [osd-admins][alert] openqaworker8: Memory usage alertResolvedokurz2020-10-122020-10-23

Actions
Copied to openQA Infrastructure (public) - action #93961: Add redundancy for SAP multi machines tests - extend RAM on machinesResolvednicksinger2021-04-082021-06-22

Actions
Actions
Copy link

Also available in: Atom PDF