Actions
action #162293
opencoordination #157969: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui and production workloads, to openSUSE Leap 15.6
SMART errors on bootup of w31+w32, possibly more
Start date:
2024-06-14
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Observation¶
Struggling with w31+w32 that upgraded themselves to Leap 15.6 and then crashed multiple times after booting into kernel 6.4 we observed that early during bootup there were SMART errors shown. Possibly this might explain kernel crashes or might be separate errors. We downgraded to Leap 15.5 for now and took it out of production but still run as openQA worker.
Acceptance criteria¶
- AC1: w31 boots up fine without SMART errors
Steps to reproduce¶
- reboot worker31 and then follow the output on
ssh -t jumpy@qe-jumpy.prg2.suse.org "ipmitool -I lanplus -H openqaworker31.qe-ipmi-ur -U … -P … sol activate"
- observe SMART errors very early during firmware initialization
Suggestions¶
- Check the content of /var/crash and clean up after investigation
- Check the status of SMART from the running Linux system and then also the messages on bootup
- Crosscheck the SMART status on other salt controlled machines, at least observed the same on w32
- Consider replacing defective hardware
- Ensure no failed services again
- Bring back the system into production
Rollback steps¶
hostname=worker31.oqa.prg2.suse.org ssh osd "sudo salt-key -y -a $hostname && sudo salt --state-output=changes $hostname state.apply"
ssh osd "worker31.oqa.prg2.suse.org' cmd.run 'systemctl unmask rebootmgr && systemctl enable --now rebootmgr && rebootmgrctl reboot'"
Actions