QA » openQA Project » openQA Infrastructure

The system is operational right now. But it seems we need salt-states-openqa to be deployed 🤔

There are some failures in the pipelines but no logs. Running pipelines manually on master atm

Pipelines/deployment seems to have succeeded

cdywan wrote:

The system is operational right now. But it seems we need salt-states-openqa to be deployed 🤔

What do you mean with that and why? Also for that we would need #43934 first.
openqaworker7 is part of o3 and not touched by triggering any gitlab CI pipelines.

But as we reopened this ticket please keep in mind that we should look for at least two improvements.

okurz wrote:

cdywan wrote:

The system is operational right now. But it seems we need salt-states-openqa to be deployed 🤔

What do you mean with that and why? Also for that we would need #43934 first.
openqaworker7 is part of o3 and not touched by triggering any gitlab CI pipelines.

The mitigation done by @favogt consisted of manually fixing the mount points. The above salt change fixes the mount points, but the pipeline was failing at the time, so I investigated that. Note how I didn't touch the ticket state or draw any conclusion so far, I just transfered ideas and steps from IRC to a non-temporary place ;-)

cdywan wrote:

[…] The above salt change fixes the mount points, but the pipeline was failing at the time, so I investigated that

yeah but please understand that the changes in the salt repo are only applied to OSD and have no impact on O3 infrastructure unless you manually copy paste the instructions from the salt git repo to the manually maintained files on o3 machines.

okurz wrote:

cdywan wrote:

[…] The above salt change fixes the mount points, but the pipeline was failing at the time, so I investigated that

yeah but please understand that the changes in the salt repo are only applied to OSD and have no impact on O3 infrastructure unless you manually copy paste the instructions from the salt git repo to the manually maintained files on o3 machines.

Sure. But you asked why I was looking into that.

I assume fvogt applied manually the slightly different variant:

ExecStart=/bin/sh -c 'test -e /dev/md/openqa || lsblk -n | grep -v nvme | grep "/$" && mdadm --create /dev/md/openqa --level=0 --force --raid-devices=$(ls /dev/nvme?n1 | wc -l) --run /dev/nvme?n1 || mdadm --create /dev/md/openqa --level=0 --force --raid-devices=1 --run /dev/nvme0n1p3'

which should also work for now. As the machine is up since another automatic reboot today I think the original problem was resolved, hence reducing prio and setting "Feedback". Now, following our best practice of "think of at least a second improvement", @cdywan what can you think of? :)

How about documenting this in the wiki:

https://progress.opensuse.org/projects/openqav3/wiki/Wiki#Mitigation-of-boot-failure-or-disk-issues

It broke again two days ago. It seems like openqa_nvme_format.service was not failing, but got triggered in an endless loop recreating the raid and formatting it over and over again.

The mdadm --stop makes the source of var-lib-openqa.mount disappar, so systemd stopped it. The creation made it appear again, so systemd might schedule yet another start of openqa_nvme_format.service. I commented out the first ExecStart to avoid that, but it showed what the actual issue is. The unit is activated way too early due to DefaultDependencies=no:

Nov 23 10:15:36 openqaworker7 systemd[1]: systemd 234 running in system mode. (+PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN default-hierarchy=hybrid)
Nov 23 10:15:36 openqaworker7 systemd[1]: Detected architecture x86-64.
Nov 23 10:15:36 openqaworker7 systemd[1]: Set hostname to <openqaworker7>.
Nov 23 10:15:38 openqaworker7 sh[935]: grep: /proc/mdstat: No such file or directory
Nov 23 10:15:38 openqaworker7 kernel: BTRFS info (device sda1): disk space caching is enabled
Nov 23 10:15:38 openqaworker7 systemd[1]: Started Load Kernel Modules.
Nov 23 10:15:38 openqaworker7 systemd[1]: openqa_nvme_format.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 23 10:15:38 openqaworker7 systemd[1]: Failed to start Setup NVMe before mounting it.
Nov 23 10:15:38 openqaworker7 systemd[1]: Dependency failed for /var/lib/openqa.

Unfortunately the system stopped responding to input for some reason, so it had to go through another slow reboot... I added

Requires=dev-md-openqa.device
After=dev-md-openqa.device

to the unit as it only does the formatting now. After yet another reboot it's now up, AFAICT race-free.

okurz wrote:

I appreciate your efforts but that mdadm call was there for a reason.

Well, in the vast majority of boots it did more harm (break completely) than good (recreate the raid if necessary), so the current state is arguably much better.

Certainly the approach was not perfect but just removing it is changing what it was "designed" for. Yes, it's only used if the RAID does not assemble automatically. This happens if the same service definition is used on fresh installs as well as when NVMe devices change, which is what happens and will happen again.

That's only manually triggered though and happens maybe once a year at most?

I will comment the suggestion in the ticket,

Which ticket?

reopen it and hope you can bring back the mdadm call in a way that fulfills the original requirements as well as convince you that the design is not "broken" anymore.

I split the removed parts into a new openqa_nvme_create.service. It seems like the "array works" case works, but the "array needs (re)creation" case isn't tested yet.

favogt wrote:

I split the removed parts into a new openqa_nvme_create.service. It seems like the "array works" case works, but the "array needs (re)creation" case isn't tested yet.

Feel free to test this part on one of the o3 workers as well by destroying/recreating the RAID and file system on top, no problem as this is only the cache+pool of openQA workers which holds only temporary data.