QA » openQA Project » openQA Infrastructure

OSD deployment has successed.

I'm not sure how the machine booted previously. I assume we booted PXE and from there "timed out" into "boot from HDD". However, PXE reports an error now. I've created [RT-PPC #181643] and added osd-admin@suse.de as CC to address this issue.
I will try to workaround the issue by manually kexec'ing the installed system as there is no dedicated bootloader entry for the installed system (just a installation entry).

Seems like a more severe issue. I can't find the systems boot disk at all:

/ # blkid
/dev/sdm1: UUID="6c7adfd9-8aa2-45e3-abf2-e6aff8ba8721"
/dev/sdf1: UUID="6c7adfd9-8aa2-45e3-abf2-e6aff8ba8721"
/dev/sdj1: UUID="e00ee584-c968-41c0-ab80-ad3ac3b68d97"
/dev/sdc1: UUID="e00ee584-c968-41c0-ab80-ad3ac3b68d97"
/ # mount /dev/sdm1 /mnt/tmp1/
/ # mount /dev/sdf1 /mnt/tmp2/
/ # mount /dev/sdj1 /mnt/tmp3/
/ # mount /dev/sdc1 /mnt/tmp4/
/ # find /mnt/tmp* -maxdepth 1
/mnt/tmp1
/mnt/tmp1/1
/mnt/tmp1/lost+found
/mnt/tmp2
/mnt/tmp2/1
/mnt/tmp2/lost+found
/mnt/tmp3
/mnt/tmp3/tmp
/mnt/tmp3/cache.sqlite-wal
/mnt/tmp3/cache.sqlite
/mnt/tmp3/cache.sqlite-shm
/mnt/tmp3/openqa.suse.de
/mnt/tmp3/lost+found
/mnt/tmp4
/mnt/tmp4/tmp
/mnt/tmp4/cache.sqlite-wal
/mnt/tmp4/cache.sqlite
/mnt/tmp4/cache.sqlite-shm
/mnt/tmp4/openqa.suse.de
/mnt/tmp4/lost+found

[   41.796572] Btrfs loaded
[   41.797425] BTRFS: device fsid ae18adf5-d27e-4fa1-93a1-6ab55263c29d devid 1 transid 2545520 /dev/mapper/sdb1
[   41.798671] BTRFS info (device dm-5): disk space caching is enabled
[   41.798675] BTRFS: has skinny extents
[   42.663776] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
[   42.665824] BTRFS error (device dm-5): bdev /dev/mapper/sdb1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[   42.667327] BTRFS error (device dm-5): bdev /dev/mapper/sdb1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
[   42.667630] BTRFS error (device dm-5): bdev /dev/mapper/sdb1 errs: wr 2, rd 1, flush 0, corrupt 0, gen 0
[   42.667661] BTRFS error (device dm-5): bdev /dev/mapper/sdb1 errs: wr 2, rd 2, flush 0, corrupt 0, gen 0
[   42.667682] BTRFS error (device dm-5): bdev /dev/mapper/sdb1 errs: wr 2, rd 3, flush 0, corrupt 0, gen 0
[   42.667701] BTRFS error (device dm-5): bdev /dev/mapper/sdb1 errs: wr 2, rd 4, flush 0, corrupt 0, gen 0
[   42.667714] BTRFS: Transaction aborted (error -5)
[   42.667732] ------------[ cut here ]------------
[   42.667734] WARNING: at fs/btrfs/extent-tree.c:2930
[   42.667735] Modules linked in: btrfs xor zlib_inflate lzo_compress lzo_decompress raid6_pq ext4 mbcache jbd2 dm_snapshot dm_bufio dm_mod sd_mod sr_mod cdrom lpfc crc_t10dif crct10dif_generic crct10dif_common ipr
[   42.667756] CPU: 146 PID: 2668 Comm: pb-discover Not tainted 4.4.92-openpower1 #1
[   42.667758] task: c000001fc6241b40 ti: c000001fc6408000 task.ti: c000001fc6408000
[   42.667760] NIP: d000000005acb344 LR: d000000005acb340 CTR: c0000000001e76a8
[   42.667762] REGS: c000001fc640b240 TRAP: 0700   Not tainted  (4.4.92-openpower1)
[   42.667763] MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28042844  XER: 20000000
[   42.667770] CFAR: c0000000005accd8 SOFTE: 1 
[   42.667770] GPR00: d000000005acb340 c000001fc640b4c0 d000000005b7ef58 0000000000000025 
[   42.667770] GPR04: 0000000000000001 00000000000004d4 0000000000000035 74726f6261206e6f 
[   42.667770] GPR08: 0000000000000007 0000000000000001 0000000000000007 000000000001ed60 
[   42.667770] GPR12: 0000000000002200 c00000000fe9b600 c000001fdd743100 c000001fdd743eb0 
[   42.667770] GPR16: c000001fdd7439c8 c000001fdd7439b0 0000000000001000 000000000026d770 
[   42.667770] GPR20: c000001fb0b401df c000001fb3df00e0 c000001fb3df0160 c000001fb3df0170 
[   42.667770] GPR24: c000001fb3c10088 0000000000000001 0000000000000000 0000000000000000 
[   42.667770] GPR28: c000001fb3df0000 c000001fb08a0800 c000001fb3c10000 fffffffffffffffb 
[   42.667808] NIP [d000000005acb344] btrfs_run_delayed_refs+0xdc/0x3a4 [btrfs]
[   42.667818] LR [d000000005acb340] btrfs_run_delayed_refs+0xd8/0x3a4 [btrfs]
[   42.667819] Call Trace:
[   42.667828] [c000001fc640b4c0] [d000000005acb340] btrfs_run_delayed_refs+0xd8/0x3a4 [btrfs] (unreliable)
[   42.667838] [c000001fc640b590] [d000000005acc1d0] btrfs_write_dirty_block_groups+0xdc/0x248 [btrfs]
[   42.667847] [c000001fc640b650] [d000000005b52ccc] commit_cowonly_roots+0x208/0x2a8 [btrfs]
[   42.667857] [c000001fc640b6e0] [d000000005adff80] btrfs_commit_transaction+0x5c0/0xaa4 [btrfs]
[   42.667867] [c000001fc640b7b0] [d000000005ad9dd0] btrfs_commit_super+0xa0/0xac [btrfs]
[   42.667877] [c000001fc640b7e0] [d000000005add474] open_ctree+0x1a7c/0x1dd8 [btrfs]
[   42.667885] [c000001fc640b910] [d000000005ab2eb0] btrfs_remount+0xc08/0xee0 [btrfs]
[   42.667889] [c000001fc640ba30] [c00000000011838c] mount_fs+0x94/0x174
[   42.667892] [c000001fc640bac0] [c0000000001334a0] vfs_kern_mount+0x64/0x138
[   42.667900] [c000001fc640bb10] [d000000005ab2950] btrfs_remount+0x6a8/0xee0 [btrfs]
[   42.667903] [c000001fc640bc30] [c00000000011838c] mount_fs+0x94/0x174
[   42.667905] [c000001fc640bcc0] [c0000000001334a0] vfs_kern_mount+0x64/0x138
[   42.667907] [c000001fc640bd10] [c00000000013758c] do_mount+0xbcc/0xcfc
[   42.667909] [c000001fc640bdd0] [c000000000137930] SyS_mount+0x90/0xc8
[   42.667912] [c000001fc640be30] [c000000000009198] system_call+0x38/0xd0
[   42.667914] Instruction dump:
[   42.667916] 7d4048a8 7d474378 7ce049ad 40c2fff4 7c0004ac 7949f7e3 40e2001c 3c620000 
[   42.667920] e86384b8 7fe4fb78 4808d0b5 e8410018 <0fe00000> 3ca20000 e8a584c0 7fc3f378 
[   42.667924] ---[ end trace 91e6b5bb365bced2 ]---
[   42.667927] BTRFS: error (device dm-5) in btrfs_run_delayed_refs:2930: errno=-5 IO failure
[   42.668000] BTRFS warning (device dm-5): Skipping commit of aborted transaction.
[   42.668003] BTRFS: error (device dm-5) in cleanup_transaction:1746: errno=-5 IO failure
[   42.668086] BTRFS error (device dm-5): cleaner transaction attach returned -30
[   42.842396] BTRFS: open_ctree failed
[   42.850783] device-mapper: snapshots: Snapshot is marked invalid.
[   42.851754] Buffer I/O error on dev dm-8, logical block 1, async page read
[   43.020417] device-mapper: snapshots: Snapshot is marked invalid.
[   43.021087] EXT4-fs (dm-8): unable to read superblock
[   43.219871] device-mapper: snapshots: Snapshot is marked invalid.
[   43.220679] EXT4-fs (dm-8): unable to read superblock

it really looks bad on here. We might face some corruption…

okurz wrote:

you have not mentioned if you put back the salt key so I checked that but it seems to be there. I could ping -4 malbec.arch but not login over ssh. https://stats.openqa-monitor.qa.suse.de/d/4KkGdvvZk/osd-status-overview?viewPanel=70&orgId=1 shows it offline and unfortunately also with the ipmi commands specified in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls I have currenly no luck. Could you please check if you can reach it and teach me how to reliably interact with that machine? Also I suggest we only resolve this ticket after we could verify in like 3 reboots that the machine comes up fine again.

yeah, seems like we still face v6 issues but this time we receive the wrong lease for the host. Maybe this caused that you couldn't connect with ssh onto the host?
This is also the reason why it shows as offline in our monitoring as telegraf on OSD tries to reach v6 (as a (wrong) AAAA record is present). I created [RT-ADM #182595] AutoReply: Host malbec.arch.suse.de receives wrong dhcpv6 lease - please update to address this issue and add our current DUID (can be found in /var/lib/wicked/duid.xml) onto arch's dhcpd6.

I guess this was caused by me. So definitely a good catch, thanks!
The ipmi issue I could reproduce. I logged onto malbec to cold reset the BMC and now it seems to be reachable again:

selenium ~ » ipmitool -I lanplus -C 3 -H fsp1-malbec.arch.suse.de -P admin chassis power status
Chassis Power is on

Taking https://progress.opensuse.org/issues/81020#note-4 into account I'd postpone the multiple reboots for now. But I keep the ticket here open now anyway.