action #87898
closedcoordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
Add grafana alert for "broken workers" as reported by openQA
Description
Motivation¶
see #78390 and if you like https://chat.suse.de/channel/testing?msg=udQguXCPNRcAABnBg . We can have "broken" workers, which openQA reports itself.
https://openqa.suse.de/admin/workers can show this list. However we should also have an alert for unexpected "broken" workers.
Acceptance criteria¶
- AC1: broken workers within https://openqa.suse.de/admin/workers raise an alert from monitor.qa.suse.de
Updated by okurz over 3 years ago
- Status changed from Workable to In Progress
- Assignee set to okurz
Updated by okurz over 3 years ago
- Status changed from In Progress to Workable
- Assignee deleted (
okurz)
I started with this but could not find according entries in influxdb. I forgot how to properly test this again. But as we have too many tickets "in progress" I will set back to "Workable".
Updated by mkittler over 3 years ago
- Status changed from Workable to In Progress
- Assignee set to mkittler
SR: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/442
I started with this but could not find according entries in influxdb.
No entries were showing up due to permission errors. Even with --debug
this was not visible at all and I could only figure it out by guessing. (So grant select on table workers to telegraf;
fixed the problem.)
Updated by okurz over 3 years ago
mkittler wrote:
So grant select on table workers to telegraf;
fixed the problem.
ok but please include that in salt as well. See https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls#L166 and following lines. And please add an alert on the panel.
Updated by openqa_review over 3 years ago
- Due date set to 2021-02-20
Setting due date based on mean cycle time of SUSE QE Tools
Updated by mkittler over 3 years ago
Updated by mkittler over 3 years ago
Updated by okurz over 3 years ago
All three MRs are merged and are effective. Today I found that osd deployment alerts have failed in the "1m after" and "10m after" deployment alerts but not the "1h after". Can you please look into that and ensure that a deployment does not trigger the "broken" alert?
Updated by mkittler over 3 years ago
MR to fix that: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/451 (commit message contains more details)
Updated by mkittler over 3 years ago
- Status changed from In Progress to Feedback
Let's wait until the next deployment to see whether it worked.
Updated by livdywan over 3 years ago
- Due date changed from 2021-02-20 to 2021-02-26
No broken workers in the web UI or alerts on osd-admins@suse.de
that I can see. Bumping the due date so we can check again later this week. Alternatively, consider breaking a worker on purpose?
Updated by mkittler over 3 years ago
- Status changed from Feedback to Resolved
The alert hasn't fired during the deployment today although we had a few broken workers for a few minutes (< 15 minutes).