Project

General

Profile

Actions

action #139103

open

openQA Project - coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances

openQA Project - coordination #139010: [epic] Long OSD ppc64le job queue

Long OSD ppc64le job queue - Decrease number of x86_64 worker slots on osd to give ppc64le jobs a better chance to be assigned jobs size:M

Added by okurz 6 months ago. Updated 3 months ago.

Status:
Blocked
Priority:
Low
Assignee:
Category:
-
Target version:
Start date:
2023-11-04
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Currently on OSD there is a longer job queue in particular for ppc64le for multiple reasons, see #139010. One idea to decrease number of x86_64 worker slots on osd to give ppc64le jobs a better chance to be assigned jobs due to the OSD openQA instance job limit.

Acceptance criteria

  • AC1: The impact of worker instance ratio by arch/class has been verified
  • AC2: Given the openQA instance job limit is impacting the ppc64le job queue When the ratio of ppc64le/all workers has been increased Then the ppc64le job age is lower

Suggestions

  • DONE Look up current number of x86_64 and qemu ppc64le jobs assuming that we have a very low ppc64le/all ratio, e.g. many workers for qemu_x86_64 and very few for qemu_ppc64le (16 as of 2023-11-04).
  • DONE Reduce number of x86_64 qemu slots if we have "too many"
  • Monitor for the impact on qemu_ppc64le job age
  • Increase the amount of ppc64le machines and then again re-enable x86_64 machines

Rollback steps

Out of scope

  • Any code changes for the scheduler

Related issues 2 (1 open1 closed)

Related to openQA Tests - action #151382: [qe-sap] test fails in iscsi_client with unclear error message, please add "ping_size_check" from https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17817 to rule out MTU problemsNew2023-11-24

Actions
Copied from openQA Infrastructure - action #139100: Long OSD ppc64le job queue - Move nue3 power8 machines to nue2Resolvedokurz2023-11-04

Actions
Actions

Also available in: Atom PDF