action #166802
open
openQA Project (public) - coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances
openQA Project (public) - coordination #139010: [epic] Long OSD ppc64le job queue
Recover worker37, worker38, worker39 size:S
Added by okurz 7 months ago.
Updated 6 months ago.
Category:
Feature requests
Description
Motivation¶
After #139103 we should ensure that all remaining currently offline machines in PRG2 oQA infra are up and running again
Acceptance criteria¶
- AC1: All w37-w39 run OSD production jobs
- AC2: non-x86_64, non-qemu jobs are still executed and not starved out by too many x86_64
Suggestions¶
- Take care to apply the workarounds from #157975-12 to prevent accidental distribution upgrades
- Read what was done in #139103, bring up all w37-w39 again into production
- Monitor for the impact on qemu_ppc64le job age as well as other non-x86_64, non-qemu jobs
- Copied from action #139103: Long OSD ppc64le job queue - Decrease number of x86_64 worker slots on osd to give ppc64le jobs a better chance to be assigned jobs size:M added
- Related to action #157726: osd-deployment | Failed pipeline for master (worker3[6-9].oqa.prg2.suse.org) added
- Subject changed from Recover worker37, worker38, worker39 to Recover worker37, worker38, worker39 size:S
- Description updated (diff)
- Status changed from Blocked to Workable
- Assignee deleted (
okurz)
- Status changed from Workable to In Progress
- Assignee set to okurz
Applying the same approach as on #139103-29 to downgrade the firewall.
- Due date set to 2024-10-03
Setting due date based on mean cycle time of SUSE QE Tools
- Related to action #167057: Run more standard, qemu OSD openQA jobs in CC-compliant PRG2 and none in NUE2 size:S added
- Related to action #134924: Websocket server overloaded, affected worker slots shown as "broken" with graceful disconnect in workers table added
- Due date deleted (
2024-10-03)
- Status changed from In Progress to Blocked
blocked by #167057 as we need to prevent #134924 and parent ticket(s) from impacting us. After #167057 we can try to enable more worker instances from PRG2 again.
due to reports in slack (also see #167081) I've removed w37 from production as well now.
- Related to action #167081: test fails in support_server/setup on osd worker37 size:S added
I moved my previous comment to #167081 which is about worker37 failing in production which according to #166802#note-11 it shouldn't even be. We need to take care to keep tickets updated.
- Status changed from Blocked to In Progress
#167057 and #157690 are done so we can continue. I powered on w39 and accepted the salt key. Now doing sudo salt --state-output=changes -C 'G@roles:worker and G@osarch:x86_64' state.apply | grep -v Result
- Related to action #157690: Simple global limit of registered/online workers size:M added
- Status changed from In Progress to Blocked
I brought up w38 as well and did
for i in worker38 worker39; do openqa-clone-job --skip-chained-deps --repeat=60 --within-instance https://openqa.suse.de/tests/15639721 {TEST,BUILD}+=-poo166802-okurz _GROUP=0 WORKER_CLASS=$i; done
but I can already see on OSD in /var/log/openqa_scheduler: [2024-10-12T13:52:26.746206Z] [warn] [pid:24409] Failed to send data to websocket server, reason: Inactivity timeout at /usr/share/openqa/script/../lib/OpenQA/WebSockets/Client.pm line 27. so #157690 is not effective, reopened and disabled and powered down both w37+w38 again.
- Related to action #168178: Limit connected online workers based on websocket+scheduler load size:M added
- Target version changed from Ready to future
Also available in: Atom
PDF