QA » openQA Project » openQA Infrastructure

Observation¶

The "Apache Workers" and "HTTP Response" alerts fired on 2023-12-11 00:48:00 +0100 CET and 2023-12-11 00:49:00 +0100 CET respectively. The alerts were resolved 5 minutes later. So the impact is likely not very high (if there was an impact at all).

Suggestions¶

• Check impact (so far there doesn't seem to be any), maybe have a look at jobs that were executed around that time
• Adjust the alert thresholds? We rarely seem to hit that limit, though?
• Check grafana state and systemd journal from the relevant time
  • https://stats.openqa-monitor.qa.suse.de/alerting/grafana/MW025mB4z/view?orgId=1
  • https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?viewPanel=15&orgId=1&from=1702249200000&to=1702421999000
  • See if a lot of jobs were scheduled that time?
• Seems to have been a one-off so far

Out of scope¶

• Moving to nginx