Project

General

Profile

action #87898

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

Add grafana alert for "broken workers" as reported by openQA

Added by okurz 6 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Organisational
Target version:
Start date:
2021-01-18
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

see #78390 and if you like https://chat.suse.de/channel/testing?msg=udQguXCPNRcAABnBg . We can have "broken" workers, which openQA reports itself.
https://openqa.suse.de/admin/workers can show this list. However we should also have an alert for unexpected "broken" workers.

Acceptance criteria

History

#1 Updated by okurz 6 months ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz

#2 Updated by okurz 6 months ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

I started with this but could not find according entries in influxdb. I forgot how to properly test this again. But as we have too many tickets "in progress" I will set back to "Workable".

#3 Updated by okurz 6 months ago

  • Parent task changed from #78390 to #80142

#4 Updated by mkittler 6 months ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler

SR: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/442

I started with this but could not find according entries in influxdb.

No entries were showing up due to permission errors. Even with --debug this was not visible at all and I could only figure it out by guessing. (So grant select on table workers to telegraf; fixed the problem.)

#5 Updated by okurz 6 months ago

mkittler wrote:
So grant select on table workers to telegraf; fixed the problem.

ok but please include that in salt as well. See https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls#L166 and following lines. And please add an alert on the panel.

#6 Updated by openqa_review 6 months ago

  • Due date set to 2021-02-20

Setting due date based on mean cycle time of SUSE QE Tools

#9 Updated by okurz 5 months ago

All three MRs are merged and are effective. Today I found that osd deployment alerts have failed in the "1m after" and "10m after" deployment alerts but not the "1h after". Can you please look into that and ensure that a deployment does not trigger the "broken" alert?

#10 Updated by mkittler 5 months ago

MR to fix that: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/451 (commit message contains more details)

#11 Updated by mkittler 5 months ago

  • Status changed from In Progress to Feedback

Let's wait until the next deployment to see whether it worked.

#12 Updated by cdywan 5 months ago

  • Due date changed from 2021-02-20 to 2021-02-26

No broken workers in the web UI or alerts on osd-admins@suse.de that I can see. Bumping the due date so we can check again later this week. Alternatively, consider breaking a worker on purpose?

#13 Updated by mkittler 5 months ago

  • Status changed from Feedback to Resolved

The alert hasn't fired during the deployment today although we had a few broken workers for a few minutes (< 15 minutes).

#14 Updated by okurz 2 months ago

  • Due date deleted (2021-02-26)

Also available in: Atom PDF