openQA Project (public) » openQA Infrastructure (public)

One option we might be able to follow is to apply https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/monitoring/grafana/worker.json.template for the host "storage.qa.suse.de" as well. For storage of course we do not have "minion jobs" or an (openQA worker) web service but as we would not alarm about "no data" we could simply ignore these :) I added roles: storage in /etc/salt/grains on storage manually now so that we can distinguish and apply to that roles within top.sls, or we apply basic monitoring for each host and on top special openQA monitoring for only all workers. I tried to extend the mine.get statement in https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/monitoring/grafana.sls#L3 to target something like roles:worker or roles:storage but could not succeed getting something to succeed in my experiments on the command line like sudo salt -l error --no-color -C 'openqa.suse.de' mine.get 'G@roles:worker and G@roles:storage' 'nodename' 'grain'. As an alternative we copy the worker template for "storage" and delete all not relevant panels.

SR: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/507

With https://gitlab.suse.de/openqa/salt-states-openqa/-/commit/da6e34e0121f1f4f4042ef3e4687873311e6e228 the systemd services monitoring/alert should now cover the storage host as well.

merged https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/507, please monitor deployment and ensure that the dashboard will be correctly shown. Maybe you can add the "additional generic nodes" on https://monitor.qa.suse.de/d/4KkGdvvZk/osd-status-overview?orgId=1 as well

The dashboards are now shown. I've been creating https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/510 to tweak the memory alert. The "ping" alert doesn't seem to work as there's no data. The dashboard isn't using the correct hostname in the query - but even if the query is fixed there doesn't seem to be any ping-data. I'm not sure why that is the case because the telegraf config pinging actually contains these hosts and they're pingable.

With https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/513#note_327858 the generic dashboard works now. Unfortunately this doesn't contain a file system alert yet because I apparently removed that part of the dashboard when removing worker-specific parts.

SR: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/514/diffs

merged and seemingly broken completely. Now it shows a line with "Disk (0 panels)"

I would try to revert it for now because at this point it is hard to change anything within Grafana's UI: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/515

(Note that I actually took the "collapsed": false, change from Grafana's JSON so I'm really wondering why it isn't working. Maybe I forgot to add some relevant sections of the diff to the commit.)

How did you create the template in the first place? I guess you saved an existing dashboard and have replaced some values with variables. So we can simply save the dashboard again and replace these variables – if you can state how you did it :)

I did it like you've guessed but forgot to add some parts of the diff. This SR should have everything needed: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/518