Project

General

Profile

Actions
Copy link

action #137504

closed

Lots of emails of the form Switched Rack PDU: Outlet .* (openqaworker-arm-.*) .*

Added by livdywan 7 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-10-06
Due date:
% Done:

0%

Estimated time:
Tags:
infra

Description

Observation

I'm seeing a lot of emails with a subject of this format this week:

Switched Rack PDU: Outlet .* (openqaworker-arm-.*) .*

The body of those emails seems to be largely the same except for the date, time and code:

Name     : qaps06nue
Location : NUE-SRV2-A-Rack8 (https://racktables.nue
Contact  : osd-admins@suse.de
http://qaps06nue.qa.suse.de
http://10.162.0.106

Serial # : BA0736005926
Date: 10/06/2023
Time: 01:36:43
Code: 0x0F0B

They're filling up my inbox and I'm not clear what purpose they are serving.

Acceptance criteria

  • AC1: Switched Rack PDU: Outlet emails are actionable
  • AC1: Emails from Racktables are known to be handled or paused

Suggestions

  • Investigate what's causing these emails to be sent
  • Ensure we have relevant tickets for affected machines
  • See also #132437
Actions
Copy link
 #1

Updated by okurz 7 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz
Actions
Copy link
 #2

Updated by okurz 7 months ago

As openqaworker-arm-2+3 will go to NUE3 into cold redundancy we very likely won't see those machines anytime soon. I removed the according notification policies from https://monitor.qa.suse.de/alerting/routes

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1007 to update the automatic actions dashboard.

Actions
Copy link
 #3

Updated by okurz 7 months ago

  • Status changed from In Progress to Resolved

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1007 merged. I have seen no further emails and there are no related alerts firing in grafana on monitor.qa.suse.de

Actions
Copy link
 #4

Updated by okurz 7 months ago

  • Tags set to infra
Actions
Copy link

Also available in: Atom PDF