Project

General

Profile

Actions

action #134735

closed

[alert] openQA piworker openqa-piworker: host up alert

Added by okurz 10 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2023-08-28
Due date:
% Done:

0%

Estimated time:


Related issues 1 (0 open1 closed)

Related to openQA Infrastructure - action #132860: openqa-piworker is unstable and needs regular power-cycles size:MResolveddheidler2023-07-172024-02-27

Actions
Actions #1

Updated by okurz 10 months ago

  • Related to action #132860: openqa-piworker is unstable and needs regular power-cycles size:M added
Actions #2

Updated by osukup 10 months ago

piworker is connected to PDU, is there any way how to remote power cycle or it needs personal action in NUE2 basement ?

Actions #3

Updated by osukup 10 months ago

silenced alert, please unsilence when is back

Actions #4

Updated by dheidler 10 months ago

It got stuck when trying to reboot triggered by rebootmgr:

Aug 27 03:30:00 openqa-piworker rebootmgrd[1608]: rebootmgr: reboot triggered now!
…
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /boot/efi...
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /var/lib/openqa/pool...
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /var/lib/velociraptor-client...
Aug 27 03:30:19 openqa-piworker systemd[1]: boot-efi.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /boot/efi.
Aug 27 03:30:19 openqa-piworker systemd[1]: var-lib-openqa-pool.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /var/lib/openqa/pool.
Aug 27 03:30:19 openqa-piworker systemd[1]: var-lib-velociraptor\x2dclient.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /var/lib/velociraptor-client.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped target Preparation for Local File Systems.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped target Swaps.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivating swap /dev/disk/by-id/mmc-SC16G_0xb3fb4f9a-part2...
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
Aug 27 03:30:19 openqa-piworker systemd[1]: systemd-remount-fs.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Remount Root and Kernel File Systems.
Aug 27 03:30:19 openqa-piworker systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Create Static Device Nodes in /dev.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dlabel-SWAP.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-label/SWAP.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dpath-platform\x2dfe340000.mmc\x2dpart2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-path/platform-fe340000.mmc-part2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dpartuuid-d61468b0\x2d02.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-partuuid/d61468b0-02.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2did-mmc\x2dSC16G_0xb3fb4f9a\x2dpart2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-id/mmc-SC16G_0xb3fb4f9a-part2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-mmcblk0p2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/mmcblk0p2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2duuid-1a4a0b1d\x2d9080\x2d45de\x2d9de4\x2dfb837c99f39d.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-uuid/1a4a0b1d-9080-45de-9de4-fb837c99f39d.
Aug 27 03:30:19 openqa-piworker systemd[1]: lvm2-monitor.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Aug 27 03:30:19 openqa-piworker systemd[1]: Reached target System Shutdown.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target Unmount All Filesystems.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target Late Shutdown Services.
Aug 27 03:30:20 openqa-piworker systemd[1]: systemd-reboot.service: Deactivated successfully.
Aug 27 03:30:20 openqa-piworker systemd[1]: Finished System Reboot.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target System Reboot.
Aug 27 03:30:20 openqa-piworker systemd[1]: Shutting down.
Aug 27 03:30:20 openqa-piworker systemd[1]: Using hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0, device /dev/watchdog
Aug 27 03:30:20 openqa-piworker systemd[1]: Set hardware watchdog to 15s.
Aug 27 03:30:20 openqa-piworker kernel: watchdog: watchdog0: watchdog did not stop!
Aug 27 03:30:20 openqa-piworker systemd-shutdown[1]: Syncing filesystems and block devices.
Aug 27 03:30:21 openqa-piworker systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Aug 27 03:30:21 openqa-piworker haveged[508]: haveged: Stopping due to signal 15
Aug 27 03:30:21 openqa-piworker haveged[508]: haveged starting up
Aug 27 03:30:21 openqa-piworker systemd-journald[499]: Journal stopped
Actions #5

Updated by okurz 10 months ago

osukup wrote in #note-2:

piworker is connected to PDU, is there any way how to remote power cycle or it needs personal action in NUE2 basement ?

remote power cycle is possible. https://racktables.nue.suse.com/index.php?page=object&tab=default&object_id=18823 mentions "PDU-FC-B2", port 20. So one can use http://epdu-b2.qe.nue2.suse.org/ as described on https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs

Actions #6

Updated by dheidler 10 months ago

  • Status changed from New to Resolved
  • Assignee set to dheidler
  • Powercycled the rpi worker setup.
  • Performed manual reboot, which worked fine.
  • Removed alert silence.
Actions

Also available in: Atom PDF