action #134735: [alert] openQA piworker openqa-piworker: host up alert - openQA Infrastructure (public) - openSUSE Project Management Tool

Custom queries

openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (dev)
QE tools team - backlog (ready issues)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (dev)
QE tools team - non-estimated (unblocked) issues (infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #134735

closed

[alert] openQA piworker openqa-piworker: host up alert

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:

Resolved

Priority:

High

Assignee:

dheidler

Category:

Target version:

openQA Project (public) - Ready

Start date:

2023-08-28

Due date:

% Done:

Estimated time:

Tags:

alert, FC Basement, piworker

Description

Observation¶

From https://stats.openqa-monitor.qa.suse.de/d/GDopenqa-piworker/dashboard-for-openqa-piworker?orgId=1&from=1693033787505&to=1693184915402&viewPanel=65105 the host seems to be down since 2023-08-27

Related issues 1 (0 open — 1 closed)

Related to openQA Infrastructure (public) - action #132860: openqa-piworker is unstable and needs regular power-cycles size:M

Resolved

dheidler

2023-07-17

2024-02-27

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by okurz over 1 year ago

Related to action #132860: openqa-piworker is unstable and needs regular power-cycles size:M added

Actions

Copy link

Updated by osukup over 1 year ago

piworker is connected to PDU, is there any way how to remote power cycle or it needs personal action in NUE2 basement ?

Actions

Copy link

Updated by osukup over 1 year ago

silenced alert, please unsilence when is back

Actions

Copy link

Updated by dheidler over 1 year ago

It got stuck when trying to reboot triggered by rebootmgr:

Aug 27 03:30:00 openqa-piworker rebootmgrd[1608]: rebootmgr: reboot triggered now!
…
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /boot/efi...
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /var/lib/openqa/pool...
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounting /var/lib/velociraptor-client...
Aug 27 03:30:19 openqa-piworker systemd[1]: boot-efi.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /boot/efi.
Aug 27 03:30:19 openqa-piworker systemd[1]: var-lib-openqa-pool.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /var/lib/openqa/pool.
Aug 27 03:30:19 openqa-piworker systemd[1]: var-lib-velociraptor\x2dclient.mount: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Unmounted /var/lib/velociraptor-client.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped target Preparation for Local File Systems.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped target Swaps.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivating swap /dev/disk/by-id/mmc-SC16G_0xb3fb4f9a-part2...
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopping Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
Aug 27 03:30:19 openqa-piworker systemd[1]: systemd-remount-fs.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Remount Root and Kernel File Systems.
Aug 27 03:30:19 openqa-piworker systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Create Static Device Nodes in /dev.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dlabel-SWAP.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-label/SWAP.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dpath-platform\x2dfe340000.mmc\x2dpart2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-path/platform-fe340000.mmc-part2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2dpartuuid-d61468b0\x2d02.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-partuuid/d61468b0-02.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2did-mmc\x2dSC16G_0xb3fb4f9a\x2dpart2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-id/mmc-SC16G_0xb3fb4f9a-part2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-mmcblk0p2.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/mmcblk0p2.
Aug 27 03:30:19 openqa-piworker systemd[1]: dev-disk-by\x2duuid-1a4a0b1d\x2d9080\x2d45de\x2d9de4\x2dfb837c99f39d.swap: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Deactivated swap /dev/disk/by-uuid/1a4a0b1d-9080-45de-9de4-fb837c99f39d.
Aug 27 03:30:19 openqa-piworker systemd[1]: lvm2-monitor.service: Deactivated successfully.
Aug 27 03:30:19 openqa-piworker systemd[1]: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Aug 27 03:30:19 openqa-piworker systemd[1]: Reached target System Shutdown.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target Unmount All Filesystems.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target Late Shutdown Services.
Aug 27 03:30:20 openqa-piworker systemd[1]: systemd-reboot.service: Deactivated successfully.
Aug 27 03:30:20 openqa-piworker systemd[1]: Finished System Reboot.
Aug 27 03:30:20 openqa-piworker systemd[1]: Reached target System Reboot.
Aug 27 03:30:20 openqa-piworker systemd[1]: Shutting down.
Aug 27 03:30:20 openqa-piworker systemd[1]: Using hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0, device /dev/watchdog
Aug 27 03:30:20 openqa-piworker systemd[1]: Set hardware watchdog to 15s.
Aug 27 03:30:20 openqa-piworker kernel: watchdog: watchdog0: watchdog did not stop!
Aug 27 03:30:20 openqa-piworker systemd-shutdown[1]: Syncing filesystems and block devices.
Aug 27 03:30:21 openqa-piworker systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Aug 27 03:30:21 openqa-piworker haveged[508]: haveged: Stopping due to signal 15
Aug 27 03:30:21 openqa-piworker haveged[508]: haveged starting up
Aug 27 03:30:21 openqa-piworker systemd-journald[499]: Journal stopped

Actions

Copy link