Project

General

Profile

Actions

action #137600

closed

[alert] Packet loss between worker hosts and other hosts size:S

Added by jbaier_cz about 1 year ago. Updated 9 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
-
Start date:
2023-10-09
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

We had multiple occurrences of packet loss alert over the weekend

alertname         Packet loss between worker hosts and other hosts alert
grafana_folder         Salt
rule_uid         2Z025iB4km

http://stats.openqa-monitor.qa.suse.de/d/EML0bpuGk?orgId=1&viewPanel=4

Currently, the problematic ones according to the panel are:

imagetester - walter1.qe.nue2.suse.org  100%
petrol-1 - walter1.qe.nue2.suse.org     100%
sapworker1 - walter1.qe.nue2.suse.org   100%

That is a little bit weird as I manually checked the first one and it can reach each other well

walter1:~ #   ping imagetester.qe.nue2.suse.org
PING imagetester.qe.nue2.suse.org (10.168.192.249) 56(84) bytes of data.
64 bytes from imagetester.qe.nue2.suse.org (10.168.192.249): icmp_seq=7 ttl=64 time=0.326 ms

jbaier@imagetester:~>  ping walter1.qe.nue2.suse.org
PING walter1.qe.nue2.suse.org (10.168.192.1) 56(84) bytes of data.
64 bytes from walter1.qe.nue2.suse.org (10.168.192.1): icmp_seq=1 ttl=64 time=0.331 ms

Suggestions

  • Confirm when this started happening or if it's no longer an issue
  • There's no paused alerts

Related issues 3 (0 open3 closed)

Related to openQA Infrastructure (public) - action #138044: Grouped seemingly unrelated alert emails are confusing size:MRejectedokurz2023-10-09

Actions
Related to openQA Infrastructure (public) - action #138005: grafana panel "Packet loss between worker hosts and other hosts" shows more than just ping to "other hosts" and hence becomes slow and triggers redundant alerts size:MResolvednicksinger2023-10-14

Actions
Related to openQA Infrastructure (public) - action #138038: diesel+petrol missing network, IPMI still reachableResolvedokurz2023-10-16

Actions
Actions

Also available in: Atom PDF