Actions
action #134735
closed[alert] openQA piworker openqa-piworker: host up alert
Start date:
2023-08-28
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Observation¶
From https://stats.openqa-monitor.qa.suse.de/d/GDopenqa-piworker/dashboard-for-openqa-piworker?orgId=1&from=1693033787505&to=1693184915402&viewPanel=65105 the host seems to be down since 2023-08-27
Updated by okurz 8 months ago
- Related to action #132860: openqa-piworker is unstable and needs regular power-cycles size:M added
Updated by dheidler 8 months ago
It got stuck when trying to reboot triggered by rebootmgr:
Aug 27 03:30:00 openqa-piworker rebootmgrd[1608]: rebootmgr: reboot triggered now!
…
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /boot/efi...
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /var/lib/openqa/pool...
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /var/lib/velociraptor-client...
Aug 27 03:30:19 openqa-piworker systemd[1]: boot-efi.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /boot/efi.
Aug 27 03:30:19 openqa-piworker systemd[1]: var-lib-openqa-pool.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /var/lib/openqa/pool.
Aug 27 03:30:19 openqa-piworker systemd[1]: var-lib-velociraptor\x2dclient.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /var/lib/velociraptor-client.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped target Preparation for Local File Systems.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped target Swaps.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivating swap /dev/disk/by-id/mmc-SC16G_0xb3fb4f9a-part2...
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
Aug 27 03:30:19 openqa-piworker systemd[1]: systemd-remount-fs.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Remount Root and Kernel File Systems.
Aug 27 03:30:19 openqa-piworker systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Create Static Device Nodes in /dev.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dlabel-SWAP.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-label/SWAP.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dpath-platform\x2dfe340000.mmc\x2dpart2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-path/platform-fe340000.mmc-part2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dpartuuid-d61468b0\x2d02.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-partuuid/d61468b0-02.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2did-mmc\x2dSC16G_0xb3fb4f9a\x2dpart2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-id/mmc-SC16G_0xb3fb4f9a-part2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-mmcblk0p2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/mmcblk0p2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2duuid-1a4a0b1d\x2d9080\x2d45de\x2d9de4\x2dfb837c99f39d.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-uuid/1a4a0b1d-9080-45de-9de4-fb837c99f39d.
Aug 27 03:30:19 openqa-piworker systemd[1]: lvm2-monitor.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Aug 27 03:30:19 openqa-piworker systemd[1]: Reached target System Shutdown.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target Unmount All Filesystems.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target Late Shutdown Services.
Aug 27 03:30:20 openqa-piworker systemd[1]: systemd-reboot.service: Deactivated successfully.
Aug 27 03:30:20 openqa-piworker systemd[1]: Finished System Reboot.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target System Reboot.
Aug 27 03:30:20 openqa-piworker systemd[1]: Shutting down.
Aug 27 03:30:20 openqa-piworker systemd[1]: Using hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0, device /dev/watchdog
Aug 27 03:30:20 openqa-piworker systemd[1]: Set hardware watchdog to 15s.
Aug 27 03:30:20 openqa-piworker kernel: watchdog: watchdog0: watchdog did not stop!
Aug 27 03:30:20 openqa-piworker systemd-shutdown[1]: Syncing filesystems and block devices.
Aug 27 03:30:21 openqa-piworker systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Aug 27 03:30:21 openqa-piworker haveged[508]: haveged: Stopping due to signal 15
Aug 27 03:30:21 openqa-piworker haveged[508]: haveged starting up
Aug 27 03:30:21 openqa-piworker systemd-journald[499]: Journal stopped
Updated by okurz 8 months ago
osukup wrote in #note-2:
piworker is connected to PDU, is there any way how to remote power cycle or it needs personal action in NUE2 basement ?
remote power cycle is possible. https://racktables.nue.suse.com/index.php?page=object&tab=default&object_id=18823 mentions "PDU-FC-B2", port 20. So one can use http://epdu-b2.qe.nue2.suse.org/ as described on https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs
Actions