QA » openQA Project » openQA Infrastructure

Mailing list requested https://infra.nue.suse.com/Ticket/Display.html?id=121433&results=2c9585f45abca9e4c6fb768ef12f2a58

After this, the mailing list would be suscribed to the host group of the nagios monitoring

OK cool. We can be waiting, but why not doing something? :)

http://docs.grafana.org/alerting/notifications/

I see our version of grafana is 4.0+ so we can enable notifications already.

I assume setting up notifications is a matter of minutes or hrs, so it make sense to 'just do it'.

I prefer https://progress.opensuse.org/issues/18164#note-27 to be solved first. Having people running around red herring in chicken headless mode sounds like no fun to me.

However this can be done in Parallel ;)

You mean the ticket which was open a year ago?? :)
I prefer to have something done, so we can improve current imperfect solution than wait for the eternity to have a perfect solution to our problems.

I meant the comment #note-27 :), but again, this can be done in parallel :)

szarate wrote:

Having people running around red herring in chicken headless mode sounds like no fun to me.

We are talking here about directed messages to limited number of people who know what to do.
So in theory we should not have chicken headless mode. Actually we should have exactly the opposite - notifications to limited number of people, so they will be on the same page; chicken headless mode stems from people being on different pages of different books ;) so they start asking questions. We all start asking questions. As we care. And we eventually all run after the same problems understanding them differently.

We want people who care, like Marita, to be well informed and have immediate and constant feedback of the production openQA. So they could feel safe situation is monitored and under control.
We do not want people to be surprised. So we do not want situations "Ups, workers were down. We did not notice. Ups"

@marita: Adding you as a watcher. The monitoring dashboard is in place. Now we would like to have notifications alerts, so you could have immediate alerts on the situation.

@nicksinger: hmmm since current solution is "work in progress" and we might have some false alarms I would consider email alerts to be sent under certain circumstances:

• adding aggregated stats for sshd and systemd services
• perhaps per architecture?
• sending emails only in special cases: if 50% of servers reports offline status, then mail people? if ppc worker is offline, send email? You know what I mean. We should expect loads of false alarms, so having emails send only if serious problems are visible, make sense to me. Then we can fine-tune the alerts as we improve, or in fact we can by then have alerts sent form our infra system. We should discuss with Coolo

Thanks a lot for the Dashboard! That is really cool!

So we currently have the notifications already there, and all workers should be reporting data properly, but there are two issues atm:

• Out of space in the monitoring host
• Security group permissions too restrictive

Notification mailing list has been created, nothing done with it yet.