action #159318: openqa-piworker host up alert - openQA Infrastructure - openSUSE Project Management Tool

Actions

action #159318

closed

openqa-piworker host up alert

Added by livdywan 13 days ago. Updated 10 days ago.

Status:

Resolved

Priority:

Normal

Assignee:

Category:

Regressions/Crashes

Target version:

openQA Project - Ready

Start date:

2023-08-09

Due date:

% Done:

0%

Estimated time:

Tags:

worker, salt, infra, reactive work, openqa-piworker

Description

Observation¶

[FIRING:1] host_up (openqa-piworker: host up alert Generic openqa-piworker host_up_alert_openqa-piworker generic)

See https://stats.openqa-monitor.qa.suse.de/d/GDopenqa-piworker/dashboard-for-openqa-piworker?orgId=1&viewPanel=65105&refresh=1m

Acceptance criteria¶

AC1: openqa-piworker is up and running

Suggestions¶

DONE Add silence
DONE Remove from salt
DONE Recover
Improve to prevent similar future in the future

Rollback actions¶

Add back openqa-piworker to salt
Remove "openqa-piworker" silence from https://monitor.qa.suse.de/alerting/silences

Related issues 1 (0 open — 1 closed)

Actions

#1

Updated by okurz 11 days ago

Description updated (diff)
Category set to Regressions/Crashes

Actions

#3

Updated by okurz 10 days ago

Related to action #159270: openqaworker-arm-1 is Unreachable size:S added

Actions

#4

Updated by okurz 10 days ago

Assignee set to nicksinger

Actions

#5

Updated by nicksinger 10 days ago

Description updated (diff)
Status changed from New to In Progress

Machine was also affected by https://sd.suse.com/servicedesk/customer/portal/1/SD-154856 which I was able to resolve myself. Now executing the rollback steps for piworker

Actions

#6

Updated by nicksinger 10 days ago

Status changed from In Progress to Resolved

Back in salt, highstate applied (only security-sensor reports problems, see https://progress.opensuse.org/issues/159060#note-7), alert silence removed

Actions

Also available in: Atom PDF