action #129484
closedopenQA Project - coordination #110833: [saga][epic] Scale up: openQA can handle a schedule of 100k jobs with 1k worker instances
openQA Project - coordination #108209: [epic] Reduce load on OSD
high response times on osd - Move OSD workers to o3 to prevent OSD overload size:M
0%
Description
Motivation¶
OSD can be overloaded by too many jobs uploading too much data. We have more workers pending installation so the situation can become even worse. On the other side o3 sometimes runs into capacity limits when a new Tumbleweed snapshot, Leap and staging tests come together. So we should logically move some machines in NUE1-SRV1 from osd to o3.
Acceptance criteria¶
- AC1: o3 has more workers than before e.g. 1-2 extra workers
- AC2: wiki listing o3 workers and racktables is up-to-date
Suggestions¶
- Identify move candidates from NUE1-SRV1
- Follow https://progress.opensuse.org/projects/openqav3/wiki#Moving-worker-from-osd-to-o3
- Inform users
- Monitor impact on o3+osd load
- Ensure wiki and racktables is up-to-date
- Double-check that we're okay to move workers e.g. talk to Ralf Unger
Updated by okurz 11 months ago
- Copied to action #129487: high response times on osd - Limit the number of concurrent job upload handling on webUI side. Can we use a semaphore or lock using the database? size:M added
Updated by okurz 11 months ago
- Subject changed from high response times on osd - Consider reducing number of openQA worker instances to high response times on osd - Move OSD workers to o3 to prevent OSD overload
- Description updated (diff)
- Priority changed from Normal to High
- Target version changed from future to Ready
Updated by livdywan 10 months ago
- Status changed from Workable to Feedback
- Assignee set to livdywan
ipmi on worker6 is fine. Following our ticket handling process to get the zone updated as a first step, and updating the moving instructions which refer to an invalid ticket link.
Unsure what to make of this:
salt-key -y -d worker6.oqa.suse.de
The key glob 'worker6.oqa.suse.de' does not match any unaccepted keys.
Updated by okurz 10 months ago
- Due date set to 2023-06-26
I tried on OSD:
sudo salt-key -d worker6.oqa.suse.de
The following keys are going to be deleted:
Accepted Keys:
worker6.oqa.suse.de
Proceed? [N/y] n
I could reproduce your problem as non-root. Ensure to be root.
I added some details to the SD ticket and updated the wiki entry.
Updated by livdywan 10 months ago
- Status changed from Blocked to In Progress
cdywan wrote:
Works with that. Marking the ticket as blocked on the SD ticket now.
Change seems to have worked, machine disappeared from oqa.suse.de as expected - and I incidentally swapped the steps to adjust mount points and restart the network on the worker after restarting dnsmasq since there's no way to get to it before 🤓
Restart network on machine (over IMPI) using systemctl restart network and monitor in o3:journalctl -f -u dnsmasq until address is assigned, e.g.:
Kind of wondering what the point of these instructions is... the machine is already reachable at this point and the network service can be restarted without getting disconnected from SSH.
Ensure all mountpoints up
mount -a
This seems to be hanging forever... will investigate this later if I see any problems, I guess.
Updated by livdywan 10 months ago
Change root password to o3 one
I guess the warning is expected and I assume it means I did set the correct password. Can't login via SSH, though🤔
Update /etc/openqa/workers.ini with similar config as used on other workers, e.g. based on openqaworker4, example:
Also copyng EXTERNAL_VIDEO_ENCODER.
And apparently there was an explicit mention of the machine in salt which was causing pipeline failures despite the salt key having been removing early on.
systemctl disable --now auto-update.timer salt-minion telegraf
My shell's getting stuck running this 🤔
Updated by livdywan 10 months ago
cdywan wrote:
Change root password to o3 one
I guess the warning is expected and I assume it means I did set the correct password. Can't login via SSH, though🤔
This got resolved by installing openssh-server-config-rootlogin.
systemctl disable --now auto-update.timer salt-minion telegraf
My shell's getting stuck running this 🤔
Failed to get load state of salt-minion.service: Connection timed out
Updated by okurz 10 months ago
cdywan wrote:
cdywan wrote:
Change root password to o3 one
I guess the warning is expected and I assume it means I did set the correct password. Can't login via SSH, though🤔
This got resolved by installing openssh-server-config-rootlogin.
Not true. That package does not exist in Leap, see https://software.opensuse.org/package/openssh-server-config-rootlogin?search_term=openssh-server-config-rootlogin . I did the change with sed in the sshd config.
systemctl disable --now auto-update.timer salt-minion telegraf
My shell's getting stuck running this 🤔
Failed to get load state of salt-minion.service: Connection timed out
Then terminate it manually and try again. Or reboot first and try again.
Updated by livdywan 10 months ago
- Status changed from In Progress to Workable
- Assignee deleted (
livdywan)
okurz wrote:
Failed to get load state of salt-minion.service: Connection timed outThen terminate it manually and try again. Or reboot first and try again.
I did. That didn't help ;-)
Edit: My other comment update got lost. zypper -n dup --force-resolution --allow-vendor-change repeatedly got stuck trying to remove openQA-common. There was no output. I didn't manage to investigate this further whilst I still had a working connection.
I'm putting the ticket back for now whilst I can't reliably connect to any o3 workers. No tracker yet since different people report it working for them, and "it's Thursday".
Updated by okurz 10 months ago
- Status changed from Workable to In Progress
- Assignee set to okurz
the system was never rebooted and it was stuck due to NFS file handles to openqa.suse.de still open. We did an unmount -l /var/lib/openqa/share but to come to a proper clean system we just triggered a reboot anyway. To shorten further reinitialization time I tried with kexec but failed. Likely the correctly command line would be kexec -l /boot/vmlinuz --initrd=/boot/initrd --reuse-cmdline && kexec -e. I extended the NFS mount line in https://progress.opensuse.org/projects/openqav3/wiki/Wiki#Moving-worker-from-osd-to-o3 as it was missing the special arguments we found to be quite helping, e.g. noauto,nofail,retry=30,ro,x-systemd.automount,x-systemd.device-timeout=10m,x-systemd.mount-timeout=30m
So rebooted and after reboot we found that openQA worker services failed to register against openqa1-opensuse. That looked suspiciously like problems observed in #78390 regarding nscd. Restarting services with nscd openqa-worker-auto-restart@1 helped. Triggering another reboot trying to reproduce the problem.
EDIT: A reboot reproduces the problem.
Maybe related https://bugzilla.opensuse.org/show_bug.cgi?id=1149603
nicksinger disabled the nscd cache lookup with enable-cache … no in /etc/nscd.conf and then after kexec'ing openQA services immediately starting up working on jobs.
/etc/openqa/workers.ini had a wrong entry in WORKER_HOSTNAME. As WORKER_HOSTNAME should resolve automatically to a working variable value I disabled the entry in the config file and restarted openQA worker services again with systemctl restart openqa-worker-auto-restart@{1..30}. Monitoring openQA jobs, e.g.
https://openqa.opensuse.org/tests/3361924
on openqaworker6:1
Regarding the nscd problem we investigated the systemd dependencies because we suspect some problem there. with for i in aarch64 openqaworker4 openqaworker6 openqaworker7 openqaworker19 openqaworker20 qa-power8-3 rebel; do echo $i && ssh root@$i "systemd-analyze critical-chain" ; done we found that the chain for worker6 looks quite different then for other workers.
openqaworker6
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
multi-user.target @5min 25.302s
└─getty.target @5min 25.286s
└─serial-getty@ttyS1.service @5min 25.271s
└─rc-local.service @24.987s +5min 250ms
└─basic.target @24.742s
└─sockets.target @24.726s
└─xvnc.socket @24.710s
└─sysinit.target @24.002s
└─systemd-update-utmp.service @23.943s +41ms
└─auditd.service @23.735s +186ms
└─systemd-tmpfiles-setup.service @23.543s +174ms
└─local-fs.target @23.486s
└─run-user-0.mount @3min 32.216s
└─local-fs-pre.target @3.814s
└─systemd-tmpfiles-setup-dev.service @3.634s +163ms
└─kmod-static-nodes.service @3.043s +391ms
└─systemd-journald.socket
└─system.slice
└─-.slice
openqaworker7
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
multi-user.target @3min 29.136s
└─cron.service @3min 29.122s
└─postfix.service @3min 26.806s +2.293s
└─time-sync.target @3min 26.237s
└─chronyd.service @3min 25.442s +778ms
└─network.target @3min 25.410s
└─wicked.service @2min 55.170s +30.223s
└─wickedd-nanny.service @2min 55.130s +21ms
└─wickedd.service @2min 55.042s +70ms
└─openvswitch.service @2min 55.002s +18ms
└─ovs-vswitchd.service @2min 54.742s +241ms
└─ovs-delete-transient-ports.service @2min 54.694s +29ms
└─ovsdb-server.service @2min 53.368s +1.307s
└─network-pre.target @2min 53.297s
└─firewalld.service @2min 49.377s +3.895s
└─polkit.service @2min 56.485s +319ms
└─basic.target @2min 49.308s
└─sockets.target @2min 49.292s
└─iscsid.socket @2min 49.276s
└─sysinit.target @2min 48.852s
└─systemd-update-utmp.service @2min 48.809s +26ms
└─auditd.service @2min 48.618s +170ms
└─systemd-tmpfiles-setup.service @2min 48.485s +114ms
└─local-fs.target @2min 48.436s
└─var-lib-openqa-share.automount @2min 48.419s
└─var-lib-openqa.mount @2min 48.211s +171ms
└─dev-md127.device @5.980s
also found that apparmor is masked. Unmasked now and started.
Updated by okurz 10 months ago
- Related to action #78390: Worker is stuck in "broken" state due to unavailable cache service (was: and even continuously fails to (re)connect to some configured web UIs) added
Updated by mkittler 10 months ago
Because I've just got a complaint about broken jobs via #factory:opensuse.org on Matrix, I stopped all worker slots for now: systemctl disable --now openqa-worker-auto-restart@{1..30} && systemctl disable --now openqa-reload-worker-auto-restart@{1..30}.path && systemctl disable --now openqa-worker-cacheservice.service
Updated by okurz 10 months ago
uefi jobs failed like https://openqa.opensuse.org/tests/3363797
I called from o3
rsync -aHP root@openqaworker7:/usr/share/qemu/*staging* . && rsync -aHP *staging* root@openqaworker6:/usr/share/qemu/
and then to verify
openqa-clone-job --within-instance https://openqa.opensuse.org/tests/3363797 WORKER_CLASS=openqaworker6 TEST+=poo129484 BUILD= _GROUP_ID=0
- > microos-Staging:N-Staging-MicroOS-Image-ContainerHost-x86_64-BuildN.61.9-container-host@uefi-staging -> https://openqa.opensuse.org/tests/3363862
The error in https://openqa.opensuse.org/tests/3361980 is a bit hidden in log but rather explicit "ffmpeg: command not found". But the job is incomplete and properly restarted. at least that works fine. I will look into installing the missing package and restarting
Updated by okurz 10 months ago
Following https://progress.opensuse.org/projects/openqav3/wiki/#Setup-guide-for-new-machines
okurz@ariel:~> rsync -aHP *staging* root@openqaworker6:/usr/share/qemu/
and on w6:
zypper -n in ffmpeg && systemctl enable --now openqa-worker-cacheservice openqa-worker-cacheservice-minion openqa-worker-auto-restart@1 openqa-reload-worker-auto-restart@1
I started just a single worker instance now. It picked up https://openqa.opensuse.org/tests/3362077
Updated by okurz 10 months ago
ok, that caused problems because worker instance 30 did not have vtap devices configured. Previously only 29 worker instances were used. So I disabled instance 30 again. Then we realized that we need to update gre tunnel connections. So in /etc/wicked/scripts/gre_tunnel_preup.sh we added according IPv4 entries for each "other" multi-machine openQA worker.
Oh, and I did
WORKER=openqaworker6 failed_since="2023-06-15" openqa-advanced-retrigger-jobs
to retrigger recent incompletes on openqaworker6 that are not handled yet.
Checking for i in aarch64 openqaworker4 openqaworker6 openqaworker7 openqaworker19 openqaworker20 qa-power8-3 rebel; do echo $i && ssh root@$i "systemctl list-timers | grep -i openqa" ; done I find more amiss
aarch64
Mon 2023-06-19 14:00:34 CEST 3min 25s left Mon 2023-06-19 13:55:34 CEST 1min 34s ago openqa-continuous-update.timer openqa-continuous-update.service
Tue 2023-06-20 03:00:00 CEST 13h left Mon 2023-06-19 03:00:34 CEST 10h ago openqa-auto-update.timer openqa-auto-update.service
openqaworker4
Mon 2023-06-19 13:59:41 CEST 2min 32s left n/a n/a openqa-continuous-update.timer openqa-continuous-update.service
Tue 2023-06-20 03:00:00 CEST 13h left Mon 2023-06-19 13:49:17 CEST 7min ago openqa-auto-update.timer openqa-auto-update.service
openqaworker6
openqaworker7
Tue 2023-06-20 03:00:00 CEST 13h left Mon 2023-06-19 03:00:40 CEST 10h ago openqa-auto-update.timer openqa-auto-update.service
n/a n/a Mon 2023-06-19 13:55:19 CEST 1min 54s ago openqa-continuous-update.timer openqa-continuous-update.service
openqaworker19
Mon 2023-06-19 13:57:28 CEST 13s left Mon 2023-06-19 13:52:28 CEST 4min 46s ago openqa-continuous-update.timer openqa-continuous-update.service
Tue 2023-06-20 03:00:00 CEST 13h left Mon 2023-06-19 03:00:49 CEST 10h ago openqa-auto-update.timer openqa-auto-update.service
openqaworker20
Tue 2023-06-20 03:00:00 CEST 13h left Mon 2023-06-19 03:00:03 CEST 10h ago openqa-auto-update.timer openqa-auto-update.service
qa-power8-3
Tue 2023-06-20 03:00:00 CEST 13h left Mon 2023-06-19 03:00:20 CEST 10h ago openqa-auto-update.timer openqa-auto-update.service
n/a n/a Mon 2023-06-19 13:57:15 CEST 234ms ago openqa-continuous-update.timer openqa-continuous-update.service
rebel
Mon 2023-06-19 13:57:30 CEST 15s left Mon 2023-06-19 13:52:30 CEST 4min 44s ago openqa-continuous-update.timer openqa-continuous-update.service
so I enabled with zypper -n in openQA-auto-update openQA-continuous-update && systemctl enable --now openqa-{auto,continuous}-update.timer
Also the output of for i in aarch64 openqaworker4 openqaworker6 openqaworker7 openqaworker19 openqaworker20 qa-power8-3 rebel; do echo $i && ssh root@$i "ovs-vsctl show | grep 192" ; done looks clean now:
aarch64
openqaworker4
options: {remote_ip="192.168.112.17"}
options: {remote_ip="192.168.112.18"}
options: {remote_ip="192.168.112.23"}
options: {remote_ip="192.168.112.12"}
openqaworker6
options: {remote_ip="192.168.112.17"}
options: {remote_ip="192.168.112.18"}
options: {remote_ip="192.168.112.7"}
options: {remote_ip="192.168.112.12"}
openqaworker7
options: {remote_ip="192.168.112.6"}
options: {remote_ip="192.168.112.7"}
options: {remote_ip="192.168.112.17"}
options: {remote_ip="192.168.112.18"}
openqaworker19
options: {remote_ip="192.168.112.23"}
options: {remote_ip="192.168.112.12"}
options: {remote_ip="192.168.112.7"}
options: {remote_ip="192.168.112.18"}
openqaworker20
options: {remote_ip="192.168.112.12"}
options: {remote_ip="192.168.112.7"}
options: {remote_ip="192.168.112.23"}
options: {remote_ip="192.168.112.17"}
Updated by okurz 10 months ago
- Due date deleted (
2023-06-26) - Status changed from In Progress to Resolved
https://racktables.nue.suse.com/index.php?object_id=4992&page=object&tab=default updated. I guess we are good.