Project

General

Profile

Actions
Copy link

action #154426

closed

HTTP Response alert Salt alerting and autoresolving shortly size:M

Added by jbaier_cz 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
High
Assignee:
jbaier_cz
Category:
-
Target version:
openQA Project - Ready
Start date:
Due date:
% Done:

0%

Estimated time:
Tags:
alert, osd, grafana, infra, reactive work, http response

Description

Observation

From Grafana/ osd-admins@suse.de

Values
B0=13.681051591 
Labels
alertname     HTTP Response alert
grafana_folder     Salt
rule_uid     tm0h5mf4k

see https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?viewPanel=78&orgId=1&from=1706482095115&to=1706485674645

Suggestions

Related issues 1 (0 open — 1 closed)

Copied from openQA Infrastructure - action #133397: HTTP Response alert Salt alerting and autoresolving shortly size:MResolvedmkittler2023-07-26

Actions
Actions
Copy link
 #1

Updated by jbaier_cz 3 months ago

  • Copied from action #133397: HTTP Response alert Salt alerting and autoresolving shortly size:M added
Actions
Copy link
 #2

Updated by okurz 3 months ago

  • Tags changed from alert, osd, grafana, http response, infra to alert, osd, grafana, http response, infra, reactive work
  • Subject changed from HTTP Response alert Salt alerting and autoresolving shortly size:M to HTTP Response alert Salt alerting and autoresolving shortly
Actions
Copy link
 #3

Updated by okurz 3 months ago

  • Subject changed from HTTP Response alert Salt alerting and autoresolving shortly to HTTP Response alert Salt alerting and autoresolving shortly size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #4

Updated by jbaier_cz 3 months ago

  • Status changed from Workable to In Progress
  • Assignee set to jbaier_cz

Relevant logs from osd shows some network issue during the period:

Jan 29 00:14:20 openqa telegraf[1327]: 2024-01-28T23:14:20Z E! [inputs.http] Error in plugin: [url=https://openqa.suse.de/admin/influxdb/minion]: Get "https://openqa.suse.de/admin/influxdb/minion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Jan 29 00:14:20 openqa telegraf[1327]: 2024-01-28T23:14:20Z E! [inputs.http] Error in plugin: [url=https://openqa.suse.de/admin/influxdb/jobs]: Get "https://openqa.suse.de/admin/influxdb/jobs": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Not related, but maybe interesting line:

Jan 29 00:15:05 openqa openqa[6000]: [debug] Rejecting authentication for user "openqaworker4" with ip "10.168.192.181", valid key "160AA95F68C410D5", secret "E38C9451DB07468D", timestamp mismatch - check whether clocks on the local host and the web UI host are in sync

Also worth noticing, there is almost no CPU load and almost zero networking traffic during that period

Actions
Copy link
 #5

Updated by okurz 3 months ago

  • Status changed from In Progress to Workable
Actions
Copy link
 #6

Updated by jbaier_cz 3 months ago

  • Status changed from Workable to Resolved

I couldn't find any more traces. As it probably was just a minor networking issue, it seems to be solved and there is no further action to be done.

Actions
Copy link

Also available in: Atom PDF