QA (public) » openQA Project (public) » openQA Infrastructure (public)

Currently grafana is available again. Since I just recently merged a commit and the deployment ran/was fixed, I assume this might correlate with the restart of grafana on the monitor host. I remember we had a similar discussion in the past and deemed it unfeasible to hot-reload grafana. I will check if something changed or if I can find an easy workaround.

Setting due date based on mean cycle time of SUSE QE Tools

nicksinger wrote in #note-6:

next steps:

• reset the admin password with the documented one
• try if the api calls work with that user
• implement a small script or something to hook into systemd to reload the service
• add a timer to properly restart grafana nightly (reloads will eventually not catch everything I think).
• add a simple error_page 502-directive (http://nginx.org/en/docs/http/ngx_http_core_module.html#) to our monitor host explaining on a simple, static page what to do if the nightly restart failed

In yesterdays daily we were discussing about a nightly restart and decided that we only implement this if we realize the other restarts (e.g. at reboot, after package updates) are not sufficient for us. I created a reload-script now and hook it into our grafana service to make systemctl reload grafana-server possible. I'm not sure yet where to put the admin credentials but most likely they will end up either manually configured or in our pillars (have to cross-check if they are public). Also until now nothing makes use of this reload function yet so I have to go trough our states and pipelines and find places where we restart grafana and replace it with a reload-or-restart.

MR merged. But I still need to add the credentials for the admin user

PW pillar MR:

1. https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1279
2. https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/909

all merged and deployed.

13.10.2024 03:37:46 root root@monitor.qa.suse.de:

grafana-server.service on host monitor.qe.nue2.suse.org failed to start
please ssh into the host and check systemctl status grafana-server.service for potential reasons

As I have received more alert messages afterwards I assume this was transient and we should look into this message to avoid it in the future

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1288

root@monitor:~ # uptime
 18:49:56  up   2:34,  1 user,  load average: 1.13, 1.25, 1.34
root@monitor:~ # systemctl status grafana-server
● grafana-server.service - Grafana instance
     Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/grafana-server.service.d
             └─00-enable-reload.conf, 01-service-fail-mail.conf, override.conf
     Active: active (running) since Tue 2024-10-15 16:16:57 CEST; 2h 33min ago

reboot was conducted and the service came up without alerting us by mail. I purposely broke /etc/grafana/grafana.ini and restarted the service which failed and logged:

root@monitor:~ # systemctl restart grafana-server
Job for grafana-server.service failed because the control process exited with error code.
See "systemctl status grafana-server.service" and "journalctl -xeu grafana-server.service" for details.
root@monitor:~ # systemctl status grafana-server
× grafana-server.service - Grafana instance
     Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/grafana-server.service.d
             └─00-enable-reload.conf, 01-service-fail-mail.conf, override.conf
     Active: failed (Result: exit-code) since Tue 2024-10-15 18:52:58 CEST; 15s ago
   Duration: 2h 35min 51.622s
       Docs: http://docs.grafana.org
    Process: 15139 ExecStart=/usr/share/grafana/bin/grafana server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=rpm cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR} cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR} (>
   Main PID: 15139 (code=exited, status=1/FAILURE)
        CPU: 506ms

Oct 15 18:52:56 monitor systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 4.
Oct 15 18:52:56 monitor systemd[1]: Starting Grafana instance...
Oct 15 18:52:57 monitor grafana[15139]: logger=settings t=2024-10-15T18:52:57.896695559+02:00 level=error msg="failed to parse \"/etc/grafana/grafana.ini\": key-value delimiter not found: NSINGER BROKE THIS _ DELETE ME IF YOU FIND ME AND RESTART GRAFANA\n"
Oct 15 18:52:57 monitor systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Oct 15 18:52:57 monitor systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Oct 15 18:52:58 monitor systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
Oct 15 18:52:58 monitor systemd[1]: grafana-server.service: Start request repeated too quickly.
Oct 15 18:52:58 monitor systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Oct 15 18:52:58 monitor systemd[1]: Failed to start Grafana instance.
Oct 15 18:52:58 monitor systemd[1]: grafana-server.service: Triggering OnFailure= dependencies.

And I received an e-mail: So it alerts on "real" issues but keeps silent on reboots.