action #152386
Updated by livdywan 5 months ago
### Observation The "Apache Workers" and "HTTP Response" alerts fired on 2023-12-11 00:48:00 +0100 CET and 2023-12-11 00:49:00 +0100 CET respectively. The alerts were resolved 5 minutes later. So the impact is likely not very high (if there was an impact at all). ### Suggestions * Check impact (so far there doesn't seem to be any), maybe have a look at jobs that were executed around that time * Adjust the alert thresholds? We rarely seem to hit that limit, though? * Check grafana state and systemd journal from the relevant time * https://stats.openqa-monitor.qa.suse.de/alerting/grafana/MW025mB4z/view?orgId=1 * https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?viewPanel=15&orgId=1&from=1702249200000&to=1702421999000 * See if a lot of jobs were scheduled that time? * Seems to have been a one-off so far ### Out of scope * Moving to nginx thresholds