Project

General

Profile

Actions
Copy link

action #159318

closed

openqa-piworker host up alert

Added by livdywan 8 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
nicksinger
Category:
Regressions/Crashes
Target version:
openQA Project (public) - Ready
Start date:
2023-08-09
Due date:
% Done:

0%

Estimated time:
Tags:
worker, salt, infra, reactive work, openqa-piworker

Description

Observation

[FIRING:1] host_up (openqa-piworker: host up alert Generic openqa-piworker host_up_alert_openqa-piworker generic)

See https://stats.openqa-monitor.qa.suse.de/d/GDopenqa-piworker/dashboard-for-openqa-piworker?orgId=1&viewPanel=65105&refresh=1m

Acceptance criteria

  • AC1: openqa-piworker is up and running

Suggestions

  • DONE Add silence
  • DONE Remove from salt
  • DONE Recover
  • Improve to prevent similar future in the future

Rollback actions

Related issues 1 (0 open — 1 closed)

Related to openQA Infrastructure (public) - action #159270: openqaworker-arm-1 is Unreachable size:SResolvedybonatakis2024-04-19

Actions
Actions
Copy link
 #1

Updated by okurz 8 months ago

  • Description updated (diff)
  • Category set to Regressions/Crashes
Actions
Copy link
 #3

Updated by okurz 8 months ago

  • Related to action #159270: openqaworker-arm-1 is Unreachable size:S added
Actions
Copy link
 #4

Updated by okurz 8 months ago

  • Assignee set to nicksinger
Actions
Copy link
 #5

Updated by nicksinger 8 months ago

  • Description updated (diff)
  • Status changed from New to In Progress

Machine was also affected by https://sd.suse.com/servicedesk/customer/portal/1/SD-154856 which I was able to resolve myself. Now executing the rollback steps for piworker

Actions
Copy link
 #6

Updated by nicksinger 8 months ago

  • Status changed from In Progress to Resolved

Back in salt, highstate applied (only security-sensor reports problems, see https://progress.opensuse.org/issues/159060#note-7), alert silence removed

Actions
Copy link

Also available in: Atom PDF