Project

General

Profile

Actions

coordination #139010

open

coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances

[epic] Long OSD ppc64le job queue

Added by okurz about 1 year ago. Updated about 2 months ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2023-11-04
Due date:
% Done:

66%

Estimated time:
(Total: 0.00 h)
Tags:

Description

Motivation

Currently on OSD there is a longer job queue in particular for ppc64le. This seems to be due to multiple reasons:

  1. Apparently multiple demanding product requirements come together including at least BCI and kernel live-patching
  2. Multi-month shortage on PowerPC related testing due to datacenter migration, see #132140
  3. KVM on PowerPC was repeatedly described as not supported anymore and deprecated and that we should not run this anymore hence there was less priority on providing more test capacity. Though for planning of migration we already prioritized setting up more free hardware reserved for manual testing instead to use for openQA tests.

I assume many teams learned that KVM on PowerPC is much more reliable for us than PowerVM for multiple reasons hence teams have worked with the temporarily higher capacity that we could provide at least for the time of the migration. By now about 60% of former kvm@powerpc capacity are available from FC Basement lab – merely a tertiary tier mitigation applied by qe tools team. The 100% which many might see as "reference" was a temporary situation preparing for datacenter migration. The actual reference should be about 1-2 years ago which is on par with the current provided capacity.

Ideas

Multiple improvement ideas, merely brainstorming with no commitment by anyone so far to conduct any of those:

  1. Communicate to all stakeholders our monitoring dashboard https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=now-7d&to=now from which it should be clear what to expect.
  2. Move prg2e target power8 machines still in nue1 to nue2 (this was based on wrong assumptions by okurz: There are no more PowerPC machines in nue1 so this leaves "3. Move nue3 power8 machines to nue2")
  3. Move nue3 power8 machines to nue2 -> #139100
  4. Try qemu on power9 from prg2 machines, also ask buildops team
  5. Ask other teams for free ressources, also orthos
  6. Decrease testing scope
  7. Decrease test runtime
  8. Decrease test failure rate, especially unreviewed, unlabeled failures
  9. DONE Try to setup free power hardware in FC Basement, e.g. mania.qe.nue2.suse.org https://racktables.nue.suse.com/index.php?page=object&object_id=9588 -> #139271
  10. Increase number of worker instances on existing kvm@powerpc machines and monitor for stability
  11. Increase openQA instance job limit to give ppc64le jobs a better chance to run
  12. Decrease number of x86_64 worker slots on osd to give ppc64le jobs a better chance to be assigned jobs -> #139103
  13. Put more effort into PowerVM, e.g. #71794, and use PowerVM more
  14. Improve automatic decision so that products are marked as "acceptable" based on just a smaller critical subset of tests and give other not-critical tests a possibility to finish later so that longer job queue don't prevent releases.
  15. New builds are triggered despite old tests have not even finished so multiple times tests "never finish" -> Improve automation to serialize building+testing+releasing
  16. For BCI: Consider reducing polling interval when to trigger new tests so that not too many tests are scheduled

Subtasks 3 (1 open2 closed)

openQA Infrastructure (public) - action #139100: Long OSD ppc64le job queue - Move nue3 power8 machines to nue2Resolvedokurz2023-11-04

Actions
openQA Infrastructure (public) - action #139103: Long OSD ppc64le job queue - Decrease number of x86_64 worker slots on osd to give ppc64le jobs a better chance to be assigned jobs size:MResolvedokurz2023-11-04

Actions
openQA Infrastructure (public) - action #166802: Recover worker37, worker38, worker39 size:SBlockedokurz

Actions

Related issues 4 (0 open4 closed)

Related to Containers and images - action #138770: [BCI] Reduce coverage for ppc64leResolvedph03nix2023-10-31

Actions
Related to Containers and images - action #138725: [BCI] Re-enable FIPS on ppc64le and s390xResolvedpherranz2023-10-30

Actions
Related to openQA Infrastructure (public) - action #139271: Repurpose PowerPC hardware in FC Basement - mania Power8 PowerPC size:MResolvedokurz2023-09-20

Actions
Copied from openQA Tests (public) - action #136130: test fails in iscsi_client due to salt 'host'/'nodename' confusion size:MResolvedmkittler2023-09-20

Actions
Actions #1

Updated by okurz about 1 year ago

  • Copied from action #136130: test fails in iscsi_client due to salt 'host'/'nodename' confusion size:M added
Actions #2

Updated by okurz about 1 year ago

  • Project changed from 46 to openQA Project (public)
  • Category changed from Enhancement to existing tests to Support
Actions #5

Updated by ph03nix about 1 year ago

Actions #7

Updated by MDoucha about 1 year ago

KVM on PowerPC was repeatedly described as not supported anymore and deprecated and that we should not run this anymore hence there was less priority on providing more test capacity. Though for planning of migration we already prioritized setting up more free hardware reserved for manual testing instead to use for openQA tests.

Migration of maintenance tests from KVM/QEMU backend to PowerVM is blocked by missing disk image support in the respective OpenQA backend implementation. We have requested the disk image support 3 years ago (#71794).

Actions #8

Updated by okurz about 1 year ago

  • Related to action #138725: [BCI] Re-enable FIPS on ppc64le and s390x added
Actions #9

Updated by okurz about 1 year ago

  • Description updated (diff)

Conducted meeting with runger, jlausuch, hrommel, pcervinka. Updating description with additions.

I will look into 3. and 12. myself.

For 3.
https://suse.slack.com/archives/C05UHQ49B7D/p1699015046191019

(Oliver Kurz) hi guys, how feasible would it be to move 1-2 machines from NUE3 "MB" to NUE2-FC_Basement?

For 12 I followed up in #139103

Actions #10

Updated by okurz about 1 year ago

  • Description updated (diff)
Actions #11

Updated by okurz about 1 year ago

  • Tracker changed from action to coordination
  • Subject changed from Long OSD ppc64le job queue to [epic] Long OSD ppc64le job queue
  • Status changed from Feedback to In Progress
  • Parent task set to #110833
Actions #12

Updated by okurz about 1 year ago

  • Subtask #139100 added
Actions #13

Updated by okurz about 1 year ago

  • Description updated (diff)
Actions #14

Updated by okurz about 1 year ago

  • Description updated (diff)
Actions #15

Updated by okurz about 1 year ago

  • Subtask #139103 added
Actions #16

Updated by okurz about 1 year ago

  • Description updated (diff)
Actions #17

Updated by okurz about 1 year ago

  • Status changed from In Progress to New
  • Assignee deleted (okurz)
  • Target version changed from Ready to future

Two subtasks defined, rest to be followed up with. Right now SUSE QE Tools does not plan to follow up with any other of the specified tasks except for the two explicit subtasks.

Actions #18

Updated by okurz about 1 year ago

  • Related to action #139271: Repurpose PowerPC hardware in FC Basement - mania Power8 PowerPC size:M added
Actions #19

Updated by jlausuch about 1 year ago

Looking at https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=now-30d&to=now&viewPanel=12
I would say that the situation is under control.
I only see a unusual peak here: https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=1699855177271&to=1699940834622&viewPanel=12
probably due to some milestone (maybe 15-SP6), but the trend is going down now, so we should be fine.
Do you agree?

Actions #20

Updated by MDoucha about 1 year ago

jlausuch wrote in #note-19:

Looking at https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=now-30d&to=now&viewPanel=12
I would say that the situation is under control.
I only see a unusual peak here: https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=1699855177271&to=1699940834622&viewPanel=12
probably due to some milestone (maybe 15-SP6), but the trend is going down now, so we should be fine.
Do you agree?

We're still running on only 16 qemu_ppc64le worker slots. When the next batch of livepatches comes, OSD will be overloaded for 2 weeks again. So the question is: Do we get more worker slots by the end of next week, or should I reduce LTP coverage for PPC64LE kernel maintenance updates?

Actions #21

Updated by okurz about 1 year ago

I am in the process to add more worker slots, see #139271. Likely to help this week still

Actions #22

Updated by okurz about 1 year ago

  • Description updated (diff)

#139271 was resolved by bringing into production 30 more qemu_ppc64le instances from the machine mania.qe.nue2.suse.org which covers point 9.

Actions #23

Updated by okurz 3 months ago

  • Subtask #166802 added
Actions #24

Updated by okurz 3 months ago

  • Category changed from Support to Regressions/Crashes
  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from future to Ready
Actions #25

Updated by okurz about 2 months ago

  • Target version changed from Ready to future
Actions

Also available in: Atom PDF