Project

General

Profile

Actions
Copy link

action #120807

closed

[alert] openqa.suse.de - worker12.oqa.suse.de 100% packet loss due to outdated AAAA record

Added by okurz about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project (public) - Ready
Start date:
2022-11-17
Due date:
% Done:

0%

Estimated time:
Tags:
reactive work

Description

Observation

okurz@openqa:~> sudo mtr -Z 2 -r -c1 worker12.oqa.suse.de
Start: 2022-11-21T14:16:23+0100
HOST: openqa                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:113:80c0:8080::4      0.0%     1    1.0   1.0   1.0   1.0   0.0
  2.|-- 2a07:de40:a201:2::7ff      0.0%     1    0.2   0.2   0.2   0.2   0.0
  3.|-- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
okurz@openqa:~> sudo mtr -Z 2 -r -c1 worker13.oqa.suse.de
Start: 2022-11-21T14:16:38+0100
HOST: openqa                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:113:80c0:8080::4      0.0%     1    2.2   2.2   2.2   2.2   0.0
  2.|-- 2a07:de40:a201:2::7ff      0.0%     1    0.2   0.2   0.2   0.2   0.0
  3.|-- worker13.oqa.suse.de       0.0%     1    0.5   0.5   0.5   0.5   0.0
okurz@openqa:~> nslookup worker12.oqa.suse.de
Server:     10.160.0.1
Address:    10.160.0.1#53

Name:   worker12.oqa.suse.de
Address: 10.137.10.12
Name:   worker12.oqa.suse.de
Address: 2a07:de40:a203:12:31ed:aa97:d3ed:5454

okurz@openqa:~> sudo mtr -Z 2 -r -c1 2a07:de40:a203:12:31ed:aa97:d3ed:5454
Start: 2022-11-21T14:17:59+0100
HOST: openqa                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:113:80c0:8080::4      0.0%     1    0.9   0.9   0.9   0.9   0.0
  2.|-- 2a07:de40:a201:2::7ff      0.0%     1    0.2   0.2   0.2   0.2   0.0
  3.|-- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0

but using the current IPv6 address that worker12 has:

okurz@openqa:~> sudo mtr -Z 2 -r -c1 2a07:de40:a203:12:ec4:7aff:fe7a:7736
Start: 2022-11-21T14:18:36+0100
HOST: openqa                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2620:113:80c0:8080::5      0.0%     1    1.0   1.0   1.0   1.0   0.0
  2.|-- 2a07:de40:a201:2::7ff      0.0%     1    1.8   1.8   1.8   1.8   0.0
  3.|-- 2a07:de40:a203:12:ec4:7af  0.0%     1    0.7   0.7   0.7   0.7   0.0

and

 nslookup worker12.oqa.suse.de
Server:     10.160.0.1
Address:    10.160.0.1#53

Name:   worker12.oqa.suse.de
Address: 10.137.10.12
Name:   worker12.oqa.suse.de
Address: 2a07:de40:a203:12:31ed:aa97:d3ed:5454

Rollback steps

  • Unpause alert "Packet loss between worker hosts and other hosts"
Actions
Copy link
 #2

Updated by okurz about 2 years ago

  • Description updated (diff)
  • Status changed from In Progress to Blocked
Actions
Copy link
 #3

Updated by okurz almost 2 years ago

  • Project changed from 46 to openQA Infrastructure (public)
  • Status changed from Blocked to Resolved

The DNS issue was fixed. The alert about packet loss could not be unpaused due to #121282 now being the problem. Extended that ticket accordingly to unpause the alert as soon as storage.qa.suse.de is good again.

Actions
Copy link

Also available in: Atom PDF