Actions
action #133553
closedOSD deployment fails due to unreachable openqaworker1
Start date:
2023-07-31
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Observation¶
https://gitlab.suse.de/openqa/osd-deployment/-/jobs/1723688
openqaworker1.qe.nue2.suse.org:
Minion did not return. [Not connected]
ipmi-openqaworker1-ipmi.qe.nue2.suse.org sol activate:
[SOL Session operational. Use ~? for help]
Login incorrect
Give root password for maintenance
(or press Control-D to continue):
so state of system should be debugged and recovered, removed from salt for now.
Also see https://stats.openqa-monitor.qa.suse.de/alerting/grafana/host_up_alert_openqaworker1/view?orgId=1
Rollback steps¶
- Add back w1 to salt
ssh osd 'sudo salt-key -y -d openqaworker1.qe.nue2.suse.org'
Updated by nicksinger 9 months ago
Checked the machine via salt. It hangs in a systemd rescue shell with the openqa_nvme_format.service failing:
× openqa_nvme_format.service - Setup NVMe before mounting it
Loaded: loaded (/etc/systemd/system/openqa_nvme_format.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2023-07-30 03:34:58 CEST; 1 day 4h ago
Process: 1106 ExecStart=/usr/local/bin/openqa-establish-nvme-setup (code=exited, status=1/FAILURE)
Main PID: 1106 (code=exited, status=1/FAILURE)
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1106]: Waiting 10 seconds before trying again after failing due to busy device.
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1106]: Trying RAID0 creation again after timeout (attempt 2 of 10)
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1106]: Stopping current RAID "/dev/md/*openqa"
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1326]: mdadm: stopped /dev/md/openqa
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1106]: Creating RAID0 "/dev/md/openqa" on: /dev/nvme0n1 /dev/nvme0n1 /dev/nvme1n1
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1330]: mdadm: You have listed more devices (2) than are in the array(1)!
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1106]: Unable to create RAID, mdadm returned with non-zero code
Jul 30 03:34:58 openqaworker1 systemd[1]: openqa_nvme_format.service: Main process exited, code=exited, status=1/FAILURE
Jul 30 03:34:58 openqaworker1 systemd[1]: openqa_nvme_format.service: Failed with result 'exit-code'.
Jul 30 03:34:58 openqaworker1 systemd[1]: Failed to start Setup NVMe before mounting it.
Updated by nicksinger 9 months ago
Executing the script from the rescue shell seems to work:
openqaworker1:~ # /usr/local/bin/openqa-establish-nvme-setup
Current mount points (printed for debugging purposes):
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=52753444k,nr_inodes=819200,mode=755,inode64)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
/dev/md1 on / type btrfs (rw,relatime,space_cache,subvolid=28003,subvol=/@/.snapshots/7808/snapshot)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1204)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/md1 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots)
/dev/md1 on /srv type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/srv)
/dev/md1 on /tmp type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/tmp)
/dev/md1 on /home type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/home)
/dev/md1 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/usr/local)
/dev/md1 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/x86_64-efi)
/dev/md1 on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/boot/grub2/i386-pc)
/dev/md1 on /root type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/root)
/dev/md1 on /opt type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/opt)
/dev/md1 on /var type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/var)
Present block devices (printed for debugging purposes):
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part
│ └─md0 9:0 0 0B 0 md
└─sda3 8:3 0 930.5G 0 part
└─md1 9:1 0 930.5G 0 raid1 /var
/opt
/root
/boot/grub2/i386-pc
[104103.253338][ T1394] md127: detected capacity change from 0 to 2781302784
/boot/grub2/x86_64-efi
/usr/local
/home
/tmp
/srv
/.snapshots
/
nvme1n1 259:0 0 953.9G 0 disk
nvme0n1 259:1 0 372.6G 0 disk
Creating RAID0 "/dev/md/openqa" on: /dev/nvme0n1 /dev/nvme1n1
mdadm: /dev/nvme0n1 appears to be part of a raid array:
level=raid0 devices=2 ctime=Sun Jul 23 03:36:06 2023
mdadm: /dev/nvme1n1 appears to be part of a raid array:
level=raid0 devices=2 ctime=Sun Jul 23 03:36:06 2023
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/openqa started.
Status for RAID0 "/dev/md/openqa"
Personalities : [raid1] [raid0]
md127 : active raid0 nvme1n1[1] nvme0n1[0]
1390651392 blocks super 1.2 512k chunks
md0 : inactive sda2[1](S)
1057780 blocks super 1.0
md1 : active raid1 sda3[1]
975701824 blocks super 1.0 [2/1] [_U]
bitmap: 3/8 pages [12KB], 65536KB chunk
unused devices: <none>
ARRAY /dev/md/openqa metadata=1.2 name=openqaworker1:openqa UUID=aa3ae6d1:41bb55a7:44c01bc9:5e9aed9e
Creating ext2 filesystem on RAID0 "/dev/md/openqa"
mke2fs 1.46.4 (18-Aug-2021)
/dev/md/openqa contains a ext2 file system
last mounted on /var/lib/openqa on Sun Jul 23 03:38:02 2023
Discarding device blocks: done
Creating filesystem with 347662848 4k blocks and 86917120 inodes
Filesystem UUID: 4b1186fc-5b94-4fad-bac8-4f8e910e682c
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
Updated by nicksinger 9 months ago
Log excerpt from a failing attempt:
Jul 30 03:34:45 openqaworker1 systemd[1]: Starting Setup NVMe before mounting it...
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1106]: Current mount points (printed for debugging purposes):
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: tmpfs on /run type tmpfs (rw,nosuid,nodev,size=52753444k,nr_inodes=819200,mode=755,inode64)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on / type btrfs (rw,relatime,space_cache,subvolid=28003,subvol=/@/.snapshots/7808/snapshot)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1204)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on /srv type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/srv)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on /tmp type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/tmp)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on /home type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/home)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/usr/local)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/x86_64-efi)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/boot/grub2/i386-pc)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1107]: /dev/md1 on /root type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/root)
Jul 30 03:34:45 openqaworker1 openqa-establish-nvme-setup[1106]: Present block devices (printed for debugging purposes):
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: sda 8:0 0 931.5G 0 disk
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: ├─sda1 8:1 0 1M 0 part
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: ├─sda2 8:2 0 1G 0 part
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: └─sda3 8:3 0 930.5G 0 part
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: └─md1 9:1 0 930.5G 0 raid1 /var
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /opt
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /root
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /boot/grub2/i386-pc
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /boot/grub2/x86_64-efi
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /usr/local
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /home
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /tmp
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /srv
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /.snapshots
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: /
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: nvme1n1 259:0 0 953.9G 0 disk
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1156]: nvme0n1 259:1 0 372.6G 0 disk
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1106]: Creating RAID0 "/dev/md/openqa" on: /dev/nvme0n1 /dev/nvme1n1
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1235]: mdadm: cannot open /dev/nvme0n1: Device or resource busy
Jul 30 03:34:51 openqaworker1 openqa-establish-nvme-setup[1106]: Waiting 10 seconds before trying again after failing due to busy device.
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1106]: Trying RAID0 creation again after timeout (attempt 2 of 10)
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1106]: Stopping current RAID "/dev/md/*openqa"
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1326]: mdadm: stopped /dev/md/openqa
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1106]: Creating RAID0 "/dev/md/openqa" on: /dev/nvme0n1 /dev/nvme1n1
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1330]: mdadm: You have listed more devices (2) than are in the array(1)!
Jul 30 03:34:58 openqaworker1 openqa-establish-nvme-setup[1106]: Unable to create RAID, mdadm returned with non-zero code
Updated by nicksinger 9 months ago
- Status changed from New to Feedback
Updated by okurz 9 months ago
merged. A reboot stability test code snippet is available in https://progress.opensuse.org/projects/openqav3/wiki/#Best-practices-for-infrastructure-work . I improved that script and made it available now in https://github.com/os-autoinst/scripts/pull/250
Updated by nicksinger 9 months ago
- Status changed from Feedback to Resolved
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/930 to avoid hard-blocks while booting in the future if the script fails again. Manual changes to test this on openqaworker1 are reverted. Reboot stability test showed 9/9 successful boots after the changes applied which gave me enough confidence to enable the machine in salt again.
Actions