QA (public) » openQA Project (public) » openQA Infrastructure (public)

I think it will work like that:

We will get an email as soon as there is an average of >= 1.0 failed service in 5 minutes.
That means for timers that run every minute it must fail 5 times in a row within 5 minutes.

For timers that run every 10 or 15 minutes, it's enough to have one failure to get an email, so for sporadic issues that happen like once a day, we would get 2 emails - one for the critical state and one for the resolved state.

For timers that don't run every minute we could actually add a retry.

Look into the plugin if it allows us to get more details about the actually failed services.

Munin plugins just deliver keys and number values.

Maybe there would be a way by calling a wrapper script on warnings/critical instead of directly sending out the email.

Have you tried setting env.silence_active_extinfo 1 then?
Then maybe it's visible here: http://127.0.0.1:8080/munin/opensuse.org/openqa.opensuse.org/systemd_units.html

livdywan wrote in #note-14:

So it seems as though the command is never used.

• I believe you need to restart munin-node after such a config change.
• I think it needs to be contact.o3admins.command in the global config. It can't be set per plugin.

Have you looked into the logfile?

/var/log/munin/munin-node.log
2024/09/26-17:00:04 [19324] Error output from systemd_units:
2024/09/26-17:00:04 [19324]     Failed to parse signal string ""Munin".
2024/09/26-17:00:04 [19324] Service 'systemd_units' exited with status 1/0.

Can you fix the config again?

edit: I fixed it and restarted munin.
Please always check the logs

tinita wrote in #note-17:

Have you looked into the logfile?

/var/log/munin/munin-node.log
2024/09/26-17:00:04 [19324] Error output from systemd_units:
2024/09/26-17:00:04 [19324]     Failed to parse signal string ""Munin".
2024/09/26-17:00:04 [19324] Service 'systemd_units' exited with status 1/0.

Can you fix the config again?

I guess I was looking in the wrong place. The service never seemed to fail.

What I see in plugin-conf.d/munin-node:

[systemd_units]
env.failed_critical 1
command /usr/local/bin/mail-units 

The command field for a plugin is telling munin what command to run for retrieving the data. That explains why it doesn't return anything I guess.
https://guide.munin-monitoring.org/en/latest/plugin/use.html

I think for the email we have to use the contact.o3admins.command in the global config.

Btw, to look at the past data you can do

rrdtool dump /var/lib/munin/opensuse.org/openqa.opensuse.org-systemd_units-failed-g.rrd

Unfortunately I couldn't find a way to trigger commands in my test container. Not finding anything from reading https://guide.munin-monitoring.org/en/latest/tutorial/alert.html#alert-variables-example-usage and https://guide.munin-monitoring.org/en/latest/reference/munin.conf.html I tried to run munin-cron (what upstream docs call munin-update) unsuccessfully. Eventually I attempted to test it on o3 (after changing the mail address temporarily) assuming the command is receiving data via stdin but again couldn't easily see why the script wouldn't work.

Maybe someone else wants to give it a try? In particular 1) clarifying exactly how "command" is executed and 2) how to test the "command" analoguous to munin-run.
Otherwise I'm starting to think I could have written a simple systemd service to send emails in less time than it is taking me to guess how to do this with munin.

To test the alert command without waiting for a status change, one can run

su - munin
/usr/lib/munin/munin-limits --always-send critical

For a script example see:
https://guide.munin-monitoring.org/en/latest/tutorial/alert.html#alerts-to-or-through-external-scripts
https://guide.munin-monitoring.org/en/latest/tutorial/alert.html#reformatting-the-output-message

(I saw the docs mentioned munin-limits. I could not find it in the normal path, so I did rpm -ql munin | grep limits. Then I did /usr/lib/munin/munin-limits --help to figure out how to run it)

I got something working with this:

% cat munin-mail
#!/usr/bin/perl
use strict;
use warnings;
use v5.10;

my ($subject, $email) = @ARGV;
my $content = '';
while (<STDIN>) {
    $content .= $_;
}
if ($subject =~ m/systemd_units/) {
    $content .= "\n" . qx{systemctl --failed};
}

open my $pipe, '|-', 'mail', '-s', $subject, '-r', $email, $email or die "Could not open pipe: $?";
print $pipe $content;
close $pipe or die "Could not close pipe: $?";

# munin.conf
contact.o3admins.command /home/tinita/munin-mail "${var:group} ${var:host} ${var:plugin} ${var:graph_category} '${var:graph_title}'" o3-admins@opensuse.org

tested with

/usr/lib/munin/munin-limits --always-send critical --config /home/tinita/munin-test.conf  --stdout --contact o3admins

/usr/lib/munin/munin-limits --always-send critical --config /home/tinita/munin-test.conf  --stdout --contact o3admins

This gets me Can't open /var/log/munin/munin-limits.log (Permission denied) at /usr/lib/perl5/vendor_perl/5.26.1/Log/Log4perl/Appender/File.pm line 151. 😶

You need to run it as user munin

tinita wrote in #note-29:

You need to run it as user munin

sudo -u munin /usr/lib/munin/munin-limits --always-send critical --stdout --contact o3admins works. Except it sends me openqa_minion_jobs instead of systemd_status.

I checked via config as well as munin-update.log and once more I stop seeing emails after it seemed to work once...

livdywan wrote in #note-30:

tinita wrote in #note-29:

You need to run it as user munin

sudo -u munin /usr/lib/munin/munin-limits --always-send critical --stdout --contact o3admins works. Except it sends me openqa_minion_jobs instead of systemd_status.

That's because openqa_minion_jobs is currently critical. So this is expected.

How about we have a session together next week? I have the impression there are a lot of misunderstandings.

One reason I see is that according to

munin-run  systemd_units
...
failed.value 0

there is no failed service
but

munin-run  systemd_status
...
failed.value 1

Compare http://127.0.0.1:8080/munin/opensuse.org/openqa.opensuse.org/systemd_units.html and http://127.0.0.1:8080/munin/opensuse.org/openqa.opensuse.org/systemd_status.html

So first you have to make sure that the plugin actually is critical (or warning, then adjust the --always-send option).

Was that plugin ever reporting a failed service?
According to

rrdtool dump /var/lib/munin/opensuse.org/openqa.opensuse.org-systemd_units-failed-g.rrd | less

It always reported 0 or NaN.
Maybe check how the plugin is retrieving its values.

I will trigger a munin-limits run now to see if we get an email about the critical service now.

I got this email, hopefully you too:

Date: Fri, 04 Oct 2024 15:52:47 +0000                                                                                                                                                                         
From: o3-admins@opensuse.org                                                                                                                                                                                  
To: o3-admins@opensuse.org                                                                                                                                                                                    
Subject: opensuse.org openqa.opensuse.org openqa_minion_jobs minion 'Minion Jobs - see https://openqa.opensuse.org/minion/jobs?state=failed'                                                                  

opensuse.org :: openqa.opensuse.org :: Minion Jobs - see https://openqa.opensuse.org/minion/jobs?state=failed
        CRITICALs: failed is 655.00 (outside range [:500]).

So it's calling our script, and this works.

Subject: opensuse.org openqa.opensuse.org openqa_minion_jobs minion 'Minion Jobs - see https://openqa.opensuse.org/minion/jobs?state=failed'

opensuse.org :: openqa.opensuse.org :: Minion Jobs - see https://openqa.opensuse.org/minion/jobs?state=failed
CRITICALs: failed is 655.00 (outside range [:500]).

So it's calling our script, and this works.

Sure. Except it's not picking up the failing systemd service:

sudo -u munin munin-run systemd_units config | grep ritical           
failed.critical 1

livdywan wrote in #note-36:

Sure. Except it's not picking up the failing systemd service:

sudo -u munin munin-run systemd_units config | grep ritical           
failed.critical 1

Well, that's the config output.
It tells munin which value of failed would make it critical.

The actual values you can get with

sudo munin-run systemd_units
...
failed.value 0

See also https://progress.opensuse.org/issues/166739#note-22

One problem seems to be that the service is displayed as loaded failed failed and the plugin only checks for the first one.

systemctl --no-pager --no-legend --all | grep run-r
● run-r3bd32618339d4da693f8aa9a4cb8cb48.service                                                                  loaded    failed   failed    /usr/bin/systemctl start abcdef.service

The command the plugin uses is

systemctl --no-pager --no-legend --all | awk '{print $1, $3}'

I'm not sure what the three columns mean exactly. The other plugin seems to detect it, so maybe we can use that one or both:

-if ($subject =~ m/systemd_units/) {
+if ($subject =~ m/systemd_(units|status)/) {

Looking at the munin emails from Sep 15 I also see that those came from "systemd services", and that's the title for the systemd_status plugin.

Subject: Munin - processes systemd services - opensuse.org :: openqa.opensuse.org

I made that change to the script in /usr/local/bin/munin-mail.
We probably would have to wait until the service is ok and then failed again to get an email.
Munin only sends emails when the status changes. Unless you configure it to always send, e.g.

contact.o3admins.always_send warning critical

I will change that now to see if we get an email and then comment it out again.

Ok, I edited munin.conf, restarted munin-node.service, and we got two emails.

Subject: opensuse.org openqa.opensuse.org systemd_status processes 'systemd services'                                                                                                                         

opensuse.org :: openqa.opensuse.org :: systemd services
        WARNINGs: Services in failed state is 1.00 (outside range [0:0]).


  UNIT                                          LOAD   ACTIVE SUB    DESCRIPTION
● run-r3bd32618339d4da693f8aa9a4cb8cb48.service loaded failed failed /usr/bin/systemctl start abcdef.service

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed.

That looks good, I'd say.
I commented it out again.

Draft: https://github.com/os-autoinst/openQA/pull/5979 Add munin alert email wrapper