action #99288
closed
[Alerting] openqaworker-arm-5 and openqaworker-arm-4: host up alert on 2021-09-26 size:M
Added by Xiaojing_liu over 2 years ago.
Updated over 2 years ago.
Description
Observation¶
use ipmitool -I lanplus -C 3 -H ipmi.openqaworker-arm-4.qa.suse.de -U admin -P password sol activate to connect arm-4, there is nothing shown.
use ipmitool to connect openqaworker-arm-5, it shows
Give root password for maintenance
(or press Control-D to continue): sulogin: cannot read /dev/ttyAMA0: Operation not permitted
[ 9426.877540] systemd-fstab-generator[3152]: x-systemd.device-timeout ignored for openqa.suse.de:/var/lib/openqa/share
Acceptance criteria¶
- AC1: openqaworker-arm-4 is up and can be rebooted without problems multiple times
- AC2: Same for openqaworker-arm-5
Suggestions¶
- DONE: Pause alerts
- Investigate problem
- Ensure reboot-stability with multiple reboots
Rollback steps¶
- Unpause alerts for arm-4 and arm-5
Workaround¶
used ipmitool power cycle to reboot arm-4 and arm-5
- Project changed from openQA Project to openQA Infrastructure
- Target version set to Ready
- Description updated (diff)
- Priority changed from Normal to Urgent
Xiaojing_liu wrote:
Workaround¶
used ipmitool power cycle to reboot arm-4 and arm-5
I can not confirm this worked. The alerts in OSD are still active. ssh openqaworker-arm-4.qa asks me for a password which it should not. Updated description with suggestions and rollback steps. Paused alerts in grafana.
I assume in #90275 the intended target state was never verified so likely no reboot after a salt high state was conducted. Right now on openqaworker-arm-5.qa it seems that /var/lib/openqa can not be correctly mounted assuming /dev/md/openqa which does not exist
- Related to action #90275: Replacement openQA OSD aarch64 hardware (was: Dedicated non-rpi aarch64 hardware for manual testing) added
openqaworker-arm-4 seems to repeatedly boot into installer medium.
openqaworker-arm-5 could be temporarily recovered by disabling the line /dev/md/openqa /var/lib/openqa ext2 defaults 0 0 in /etc/fstab, calling mount -a and logging out of the recovery shell which continued the boot.
@nicksinger didn't you mention necessary changes to support the filesystem setup slightly differing from other hosts?
- Description updated (diff)
- Subject changed from [Alerting] openqaworker-arm-5 and openqaworker-arm-4: host up alert on 2021-09-26 to [Alerting] openqaworker-arm-5 and openqaworker-arm-4: host up alert on 2021-09-26 size:M
- Description updated (diff)
- Status changed from New to Workable
- Assignee set to nicksinger
- Target version deleted (
Ready)
- Description updated (diff)
- Status changed from Workable to In Progress
- Target version set to Ready
- Target version deleted (
Ready)
Set the bootdev for openqaworker-arm-4 persistent with: ipmitool -I lanplus -C 3 -H ipmi.openqaworker-arm-4.qa.suse.de chassis bootdev disk options=persistent. Reboot worked 4/4 times. Another highstate also confirmed that the fstab-entry does not reappear. Doing the reboot-test now with arm-5.
- Status changed from In Progress to Resolved
- Target version set to Ready
gnah sorry for deleting "Ready" once again… openqaworker-arm-5 also survived 3/3 reboots. I consider this ticket covered then.
- Status changed from Resolved to Feedback
waaait … what happens if salt overwrites fstab changes again? I think the AC should be "survives multiple reboot+salt+reboot cycles" :) Or is the above really that you reverted parts but never rebooted so missed the fstab part?
- Status changed from Feedback to Resolved
okurz wrote:
waaait … what happens if salt overwrites fstab changes again? I think the AC should be "survives multiple reboot+salt+reboot cycles" :) Or is the above really that you reverted parts but never rebooted so missed the fstab part?
Right, I reverted the changes manually after applying a proper fix in salt (the addition of another grain) and never rebooted afterwards. In https://progress.opensuse.org/issues/99288#note-10 I mentioned another highstate to confirm this is working. Another way to view this: arm-5 was deployed first and showed problems while arm-4 didn't have the fstab problems (it was deployed with the grain already present) but just a wrong boot order :)
So your newly formulated AC is met as well
Also available in: Atom
PDF