action #97943
closed
openQA Project - coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
Increase number of CPU cores on OSD VM due to high usage size:S
Added by okurz over 2 years ago.
Updated over 2 years ago.
Description
Motivation¶
With number of workers and parallel processed tests as well as with the increased number of products tested on OSD and users using the system the workload on OSD constantly increases. CPU load alerts had been seen recently in #96713 and the higher load is visible in https://monitor.qa.suse.de/d/WebuiDb/webui-summary?viewPanel=25 . We should increase the number of CPU cores on the OSD VM due to the higher usage.
Acceptance criteria¶
- AC1: A reasonable higher number than the current 10 cores are available on OSD
Suggestions¶
- Copied from action #96713: Slow grep in openqa-label-known-issues leads to high CPU usage added
- Subject changed from Increase number of CPU cores on OSD VM due to high usage to Increase number of CPU cores on OSD VM due to high usage size:S
- Description updated (diff)
- Status changed from New to Workable
- Status changed from Workable to Feedback
repasting from SD, as i see you can't access it:
it's currently running on morla11 and it has quite constant cpu usage around 50%.
the problem might be that most morlas have 48cores, but one only on 32, so if during maintenance this workload ends up on the weaker one, we're already at 75%, so there won't be much overhead for spikes. If we want to have such huge VM, we'll need to make some special handling for it to be very sure it won't affect other workload. Does it definitely need to be one huge node or can you have multiple smaller workers distributed more evenly over the cluster?
As mentioned in the Jira ticket it currently needs to be one host. I suppose we need to decide in the team how we'd like to proceed here.
Discussed it in the weekly. From what I understand:
- We can't get more CPU cores
- Additional VMs could be provided
- Dedicated hardware instead of a VM could also be provided
@mkittler please ask if we can get shared storage between multiple VMs
dedicated HW is usually bought by the team itself. I can discuss this with Evzenia when she comes back from vacation
it is technically possible to have a NFS share, if you need shared storage, if that helps you having more smaller VMs. how much would it need?
- Status changed from Feedback to Resolved
I'm resolving this ticket with the outcome that it is not possible to extend the VM. It would be possible to request another VM so I created #99549 to split our workload.
- Related to coordination #99549: [epic] Split production workload onto multiple hosts (focusing on OSD) added
- Parent task set to #80142
Also available in: Atom
PDF