Project

General

Profile

Actions

coordination #158110

open

coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances

[epic] Prevent worker overload

Added by okurz 9 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
QA (public, currently private due to #173521) - future
Start date:
2024-03-27
Due date:
% Done:

57%

Estimated time:
(Total: 0.00 h)

Description

Motivation

For long overloaded openQA workers can cause "typing issues" and other random issues causing annoying sporadic test failures unless openQA worker machines are carefully configured to not be overloaded which is normally done with running just a limited number of worker instances accounting for cases when all worker instances work on rather resource heavy jobs. As a consequence in most cases openQA hardware can be severly underused. To make more efficient use of hardware resources while keeping openQA jobs as stable as possible openQA must ensure itself that resources are not exhausted.

Ideas

  • Only pick up (or start) new jobs if CPU load is below configured threshold -> #158125
  • Overload openQA systems on purpose to find out which system parameters are critical and define according feature requests for each relevant system parameter to be automatically handled by openQA, e.g. check CPU load, available memory, I/O rate, storage space, etc.
    • before starting jobs
    • while running jobs
    • after jobs failed

Subtasks 7 (3 open4 closed)

openQA Infrastructure (public) - action #158104: typing issue on ppc64 worker size:SResolvedokurz2024-03-27

Actions
openQA Infrastructure (public) - action #158113: typing issue on ppc64 worker - make CPU load alert more strict size:MResolvedokurz2024-03-27

Actions
openQA Infrastructure (public) - action #158116: typing issue on ppc64 worker - crosscheck performance impact of ffmpeg on ppc64le (Power8 kvm) size:MWorkable2024-03-27

Actions
action #158125: typing issue on ppc64 worker - only pick up (or start) new jobs if CPU load is below configured threshold size:MResolvedmkittler

Actions
openQA Infrastructure (public) - action #158709: typing issue on ppc64 worker - with automatic CPU load based limiting in place let's increase the instances on mania againNew

Actions
action #158910: typing issue on ppc64 worker - reconsider number of worker instances in particular on ppc64le kvm tests size:MBlockedokurz

Actions
action #168244: reconsider load calculation for worker load limit especially for ppc size:SResolvedokurz

Actions
Actions

Also available in: Atom PDF