QA » openQA Project » openQA Infrastructure

https://infra.nue.suse.com/SelfService/Display.html?id=63262 about monitoring of the workers has just been resolved. I fail to login to icinga right now. Anyone can check?

Also salt-minion

Discussing with coolo regarding this issue, updated summary:

1) Focus initially on one worker, and then replicate to the rest.
2) Focus on:

• That the worker is reachable
• SSH status
• salt-minion status
• openqa services status (os-autoinst-openvswitch, openqa-worker@*, etc)
• Core dumps
• openqa jobs assigned to workers vs scheduled

3) Review grafana as an alternate monitoring/report tool
4) IT-managed nagios instance could solve some of the metrics/services to focus on

https://wiki.microfocus.net/index.php?title=SUSE-Development/OPS/Services/Monitoring

acarvajal wrote:

3) Review grafana as an alternate monitoring/report tool

grafana and its stack is actually more suited for performance profiling. Newer versions indeed include some monitoring capabilities but IMHO it's way more efficient to use icinga/nagios for monitoring.

acarvajal wrote:

4) IT-managed nagios instance could solve some of the metrics/services to focus on

Definitely the way to go if you really just want to get some kind of monitoring. Especially the basic scenarios like ping-check, ssh-check, disk-check and so on are already covered in the basic modules provided by infra/icinga/nagios.

Requesting access to host group 'openqa-suse' on thruk with https://infra.nue.suse.com/SelfService/Display.html?id=109884

New ticket created specifically to request access to openqa-suse host group: https://infra.nue.suse.com/Ticket/Display.html?id=110564

Created 2 subtasks to tackle independently:

(1) The monitoring of the workers via the existing monitoring platform in SUSE
(2) A proof-of-concept with grafana/graphite for performance profiling

Also add to monitoring list: Threshold of jobs scheduled vs free workers

Updated sub-task with systemd timer information and salt proposal: https://progress.opensuse.org/issues/35536

One thing to note is that the dashboard is not always reliable, for this we need two tickets from infra to be solved as expressed in poo#41336 and the get-metrics script has to be moved to collectd to help poo#35536 to move forward and have better reaction and monitoring capabilities

• https://infra.nue.suse.com/Ticket/Display.html?id=121425
• https://infra.nue.suse.com/Ticket/Display.html?id=121433

infra reported that they can't provide a vm in the current cloud: https://infra.nue.suse.com/Ticket/Display.html?id=121613 steven mentioned that Max would take a look at this and propose a different solution

• fqdn requested for the current grafana instance: monitoring.openqa.suse.de -> openqa-monitoring.suse.de.
• Capacity of the grafana instance stays at ~50GB but now collectd data takes 3.5GB for all workers (only x86 for the time being due to collectd not being available on other platforms).
• still no ETA on the nagios monitoring ticket

I don't think it's blocked anymore, nicksinger, WDYT?

Thinking of monitoring systemctl --failed. As suggested by coolo: one can run commands repeatedly and check their exit status to monitor, small shell snippets in telegraf will do. https://github.com/influxdata/telegraf/tree/master/plugins/inputs/exec . so count systemctl --failed | wc -l and spit it as influxdb value.