Actions
action #123025
closedo3 worker openqaworker4 is down; boots to emergency shell only
Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2023-01-12
Due date:
2023-01-27
% Done:
0%
Estimated time:
Tags:
Description
Observation¶
When checking IPMI access (for #120270) I've noticed that openqaworker4 booted into the emergency shell. A reboot didn't help. At least IPMI access works (via jumpy@qe-jumpy.suse.de
, as documented in pillars).
I've got the following over SOL:
$ [(582a0875e...)] ssh -4 jumpy@qe-jumpy.suse.de -- ipmitool -I lanplus -C 3 -H openqaworker4-ipmi.qe-ipmi-ur -U … -P … sol activate
tcgetattr: Inappropriate ioctl for device
[SOL Session operational. Use ~? for help]
g: dracut-initqueue: starting timeout scripts
[ 154.763096] dracut-initqueue[686]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[ 154.872232] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62.sh: "[ -e "/dev/disk/by-id/md-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62" ]"
[ 154.926722] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-c807d4f1:469952f3:a2a7bf78:b5ae26ed.sh: "[ -e "/dev/disk/by-id/md-uuid-c807d4f1:469952f3:a2a7bf78:b5ae26ed" ]"
[ 155.040172] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-uuid\x2fb16b75b5-fff1-4bf3-9c31-dad4c807a49a.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
[ 155.042089] dracut-initqueue[686]: [ -e "/dev/disk/by-uuid/b16b75b5-fff1-4bf3-9c31-dad4c807a49a" ]
[ 155.096575] dracut-initqueue[686]: fi"
[ 155.270418] dracut-initqueue[686]: Warning: dracut-initqueue: starting timeout scripts
[ 155.327020] dracut-initqueue[686]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[ 155.385155] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62.sh: "[ -e "/dev/disk/by-id/md-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62" ]"
[ 155.441888] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-c807d4f1:469952f3:a2a7bf78:b5ae26ed.sh: "[ -e "/dev/disk/by-id/md-uuid-c807d4f1:469952f3:a2a7bf78:b5ae26ed" ]"
[ 155.443048] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-uuid\x2fb16b75b5-fff1-4bf3-9c31-dad4c807a49a.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
[ 155.499042] dracut-initqueue[686]: [ -e "/dev/disk/by-uuid/b16b75b5-fff1-4bf3-9c31-dad4c807a49a" ]
[ 155.556320] dracut-initqueue[686]: fi"
…
[ 203.066174] dracut-initqueue[686]: Warning: dracut-initqueue: starting timeout scripts
[ 203.066203] dracut-initqueue[686]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[ 203.066230] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62.sh: "[ -e "/dev/disk/by-id/md-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62" ]"
[ 203.066257] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-c807d4f1:469952f3:a2a7bf78:b5ae26ed.sh: "[ -e "/dev/disk/by-id/md-uuid-c807d4f1:469952f3:a2a7bf78:b5ae26ed" ]"
[ 203.066284] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-uuid\x2fb16b75b5-fff1-4bf3-9c31-dad4c807a49a.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
Starting Dracut Emergency Shell...
[ 210.752284] dracut-initqueue[686]: [ -e "/dev/disk/by-uuid/b16b75b5-fff1-4bf3-9c31-dad4c807a49a" ]
[ 210.752327] dracut-initqueue[686]: fi"
[ 210.752349] dracut-initqueue[686]: Warning: dracut-initqueue: starting timeout scripts
[ 210.752369] dracut-initqueue[686]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[ 210.752400] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62.sh: "[ -e "/dev/disk/by-id/md-uuid-b22a1df4[ 222.367453] dracut-initqueue[686]: [ -e "/dev/disk/by-uuid/b16b75b5-fff1-4bf3-9c31-dad4c807a49a" ]
[ 222.367484] dracut-initqueue[686]: fi"
…
[ 222.713160] dracut-initqueue[686]: Warning: dracut-initqueue: starting timeout scripts
[ 222.713181] dracut-initqueue[686]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[ 222.713201] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62.sh: "[ -e "/dev/disk/by-id/md-uuid-b22a1df4:ee1e951a:46cd5205:209e6a62" ]"
[ 222.713219] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-c807d4f1:469952f3:a2a7bf78:b5ae26ed.sh: "[ -e "/dev/disk/by-id/md-uuid-c807d4f1:469952f3:a2a7bf78:b5ae26ed" ]"
[ 222.713260] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-uuid\x2fb16b75b5-fff1-4bf3-9c31-dad4c807a49a.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
[ 222.713284] dracut-initqueue[686]: [ -e "/dev/disk/by-uuid/b16b75b5-fff1-4bf3-9c31-dad4c807a49a" ]
[ 222.713302] dracut-initqueue[686]: fi"
[ 222.713320] dracut-initqueue[686]: Warning: dracut-initqueue: starting timeout scripts
[ 222.713348] dracut-initqueue[686]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[ 223.045600] dracut-initqueue[686]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdi
Login incorrect
Give root password for maintenance
(or press Control-D to continue): …
sh-4.4#
Acceptance criteria¶
- AC1: openqaworker4 is back online
Actions