QA » openQA Project » openQA Infrastructure

Just a note on how to find out about failing hook scripts manually, as we have the hook_rc now:

select id, args, notes->'hook_rc', notes->'hook_result', created, finished from minion_jobs where cast(notes->'hook_rc' as int)  != 0 order by created;

tinita wrote:

Just a note on how to find out about failing hook scripts manually, as we have the hook_rc now:

select id, args, notes->'hook_rc', notes->'hook_result', created, finished from minion_jobs where cast(notes->'hook_rc' as int)  != 0 order by created;

But be aware that this query would use a sequential scan, so maybe don't run it too frequently:

                                                       QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=8101.87..8203.47 rows=40638 width=146) (actual time=52.812..53.434 rows=2092 loops=1)
   Sort Key: created
   Sort Method: quicksort  Memory: 2221kB
   ->  Seq Scan on minion_jobs  (cost=0.00..4990.92 rows=40638 width=146) (actual time=0.061..47.925 rows=2092 loops=1)
         Filter: (((notes -> 'hook_rc'::text))::integer <> 0)
         Rows Removed by Filter: 39143
 Planning Time: 0.156 ms
 Execution Time: 53.651 ms
(8 rows)

btw this is the "telegraf" route https://openqa.opensuse.org/admin/influxdb/minion

use DateTime;
$end = DateTime->now;
$end->subtract(minutes => ($dt->minute() % 10), seconds => $dt->second());
$start = $end->clone()->subtract(minutes => 10);

for the record, there is also DateTime->now->truncate( to => 'minute' ) to cut off the seconds, but if you are subtracting minutes there, it makes sense to subtract the seconds in the same call.

Also I think the query can be further reduced to hook_script tasks, e.g. where task = 'hook_script'

2do after merge: add graph in grafana including alert.

For the record: I looked into the current gru journal on o3. There are some recent grep timeouts, however, no hook scripts are marked with a nonzero exit code in the minion database.
I looked into the code to check the logic again, and currently if the search_log() call fails, the script is just returning and forgetting about the search_log exit code: https://github.com/os-autoinst/scripts/blob/master/_common#L106

We might want to change that at some point by checking on the return code in label-on-issue itself. Only the exit codes 0 (grep was successful) and 1 (grep didn't find something) are expected.

Let's wait for the deployment so we have some data to test the alert itself to be implemented.

dheidler wrote:

2do after merge: add graph in grafana including alert.

that does not make any sense as there is no grafana for o3

okurz wrote:

dheidler wrote:

2do after merge: add graph in grafana including alert.

that does not make any sense as there is no grafana for o3

AC's and suggestions include o3 and osd.

cdywan wrote:

btw this is the "telegraf" route https://openqa.opensuse.org/admin/influxdb/minion

Apparently the changes are effective:

openqa_minion_jobs,url=https://openqa.opensuse.org active=1i,delayed=0i,failed=352i,inactive=0i
openqa_minion_jobs_hook_rc_failed,url=https://openqa.opensuse.org rc_failed_per_10min=0i 16860372000000000
openqa_minion_workers,url=https://openqa.opensuse.org active=1i,inactive=0i,registered=1i

The route also works on osd:

openqa_minion_jobs,url=https://openqa.suse.de active=0i,delayed=0i,failed=3i,inactive=0i
openqa_minion_jobs_hook_rc_failed,url=https://openqa.suse.de rc_failed_per_10min=0i 16860372000000000
openqa_minion_workers,url=https://openqa.suse.de active=0i,inactive=1i,registered=1i

https://github.com/os-autoinst/openQA/pull/5190

Discussed in the Unblock:

• Why don't we use logwarn?

  • Do we always want to be alerted vs. a threshold?
  • Gru logs into the journal. That's not visible to logwarn, is it?
  • What about grafana on o3? There seems to be an instance that we weren't aware of (metrics.opensuse.org)

• Why do we care to have more advanced monitoring on o3 now?

  • We have Dimstar
  • Dimstar has no plugin to monitor investigate jobs
  • Fixing things up as needed based on user feedback is considered okay for o3
  • Back to basics - simple options
  • Forget about the ratio for o3
  • Get logwarn to pick up logs from the journal OR move logging to files?
  • Try and write a munin plugin - this could be a super simple bash script written in minutes (example plugin for minion_jobs
  • sending emails from o3 is already possible and used e.g. by logwarn called by cron, see /etc/cron.d/logwarn_openqa which means that simply cron sends any stdout by email to o3-admins@suse.de

cdywan wrote:

- Try and write a munin plugin - this could be a super simple bash script written in minutes ([example plugin for minion_jobs](https://gist.github.com/perlpunk/4aec2d85c51b22261c0717afc896aa4c)

https://github.com/os-autoinst/openQA/pull/5194

I'm fighting with getting alert emails to work with my local munin. The documentation about that is a bit confusing, and while I've written various munin plugins, I never configured email notifications myself.

Adding grafana chart: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/882

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/884

I installed the munin plugin on o3, but for now it points to a clone under my personal user, because the openQA zypper package doesn't contain the contrib folder.

Create the tunnel ssh -L 8080:localhost:80 o3 and view the data:
http://127.0.0.1:8080/munin/minion-day.html

https://github.com/os-autoinst/openQA/pull/5210 munin: Make alert thresholds configurable

https://github.com/os-autoinst/openQA/pull/5215 Add subpackage openQA-munin

Installed openQA-munin package on o3. For configuration see /etc/munin/plugin-conf.d/openqa-minion