action #119656
closedcoordination #121876: [epic] Handle openQA review failures in Yam squad - SLE 15 SP5
[Timebox: 8h] Try to increase RAM to settle the load
Description
Description¶
In openQA job, wait_serial failed to get the expected string "TOP-DONE", actually the string "TOP-DONE" was output in serial0.txt which means that we were not able to settle the load, as we can see that the load average values are really high: https://openqa.suse.de/tests/9825903#step/force_scheduled_tasks/4
Acceptance criteria¶
AC1: Increase RAM a bit more than current 4GB and check.
Suggestion¶
Let's try to increase 1024 MB to start, we have to take into account that we select all the product modules for this scenarios.
Updated by jgwang about 2 years ago
- Subject changed from wait_serial failed to get the expected string to performance issue
- Description updated (diff)
Updated by JERiveraMoya about 2 years ago
- Subject changed from performance issue to [Timebox: 8] Try to increase RAM to settle the load
- Description updated (diff)
- Status changed from New to Workable
- Target version set to Current
We found other case, but it doesn't look related because it happens in a different architecture, maybe a performance issue on the worker, but we are not performance squad, so we can just adjust RAM or other values, our goal is to have the test running, potentially file a bug for those, but they are unlikely to be taking into account with this scenarios with too many steps to reproduce, this is the other case: prepare_test_data.
restarting it in case it could help and was something temporary: https://openqa.suse.de/tests/9848854
Updated by JERiveraMoya about 2 years ago
- Subject changed from [Timebox: 8] Try to increase RAM to settle the load to [Timebox: 8h] Try to increase RAM to settle the load
Updated by jgwang about 2 years ago
- Status changed from Workable to In Progress
- Assignee set to jgwang
Updated by jgwang about 2 years ago
I increased RAM to 5G and 6G for s390x VM, these are the test results:
offline_sles12sp5_media_sdk-asmm-contm-lgm-tcm-wsm-pcm_all_full:
RAM 4G:
https://openqa.suse.de/tests/9861555#step/force_scheduled_tasks/6 ==> failed: force_sheduled_tasks
RAM 5G:
https://openqa.suse.de/tests/9861438#step/force_scheduled_tasks/6 ==> failed: force_scheduled_tasks
https://openqa.suse.de/tests/9863249#step/force_scheduled_tasks/6 ==> failed: force_sheduled_tasks
RAM 6G:
https://openqa.suse.de/tests/9861447#step/upgrade_snapshots/5 ==> failed: upgrade_snapshots
https://openqa.suse.de/tests/9863250#step/system_state/4 ==> failed: system_state
offline_sles12sp4_ltss_media_sdk-asmm-contm-lgm-tcm-wsm-pcm_all_full:
RAM 4G:
https://openqa.suse.de/tests/9861539#step/force_scheduled_tasks/9 ==> failed: force_sheduled_tasks
https://openqa.suse.de/tests/9863244#step/force_scheduled_tasks/6 ==> failed: force_sheduled_tasks
RAM 5G:
https://openqa.suse.de/tests/9861537 ==> passed
https://openqa.suse.de/tests/9863245#step/force_scheduled_tasks/6 ==> failed: force_sheduled_tasks
RAM 6G:
https://openqa.suse.de/tests/9861538#step/system_state/8 ==> failed: system_state
https://openqa.suse.de/tests/9863246#step/zypper_lr/4 ==> failed: zypper_lr
https://openqa.suse.de/tests/9865429#step/force_scheduled_tasks/6 ==> failed: force_sheduled_tasks
based on the above test results, I think this issue is not related to RAM, other factors should have caused this issue.
Updated by jgwang about 2 years ago
in the job for Build42.5 https://openqa.suse.de/tests/9970511,
the issue passed, I think it is a performance issue, and not
related to memory size. Suggest to close this ticket, but we
can open it again if this issue happens again.
Updated by JERiveraMoya almost 2 years ago
- Status changed from In Progress to Resolved
thanks for the investigation, seems that we hit some other error earlier to actually see if it improved.
Let's file a new ticket if we encounter later.