Project

General

Profile

Actions

action #176175

open

coordination #161414: [epic] Improved salt based infrastructure management

[alert] Grafana failed to start due to corrupted config file

Added by tinita about 1 month ago. Updated about 1 month ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2025-01-26
Due date:
% Done:

0%

Estimated time:

Description

Observation

From

  • Date: Sun, 26 Jan 2025 02:36:49 +0000 (UTC) to
  • Date: Sun, 26 Jan 2025 08:42:55 +0000 (UTC)

we got 10 emails

grafana-server.service on host monitor.qe.nue2.suse.org failed to start
please ssh into the host and check systemctl status grafana-server.service for potential reasons
% journalctl -u grafana-server.service --since "2025-01-26 02:30:00"
...
Jan 26 03:39:26 monitor systemd[1]: Starting Grafana instance...                   
Jan 26 03:39:26 monitor grafana[22024]: logger=settings t=2025-01-26T03:39:26.533904269Z level=error msg="failed to parse \"/etc/grafana/grafana.ini\": key-value delimiter not found: \u0000\u0000\u0000\u0000\u>
Jan 26 03:39:26 monitor grafana[22024]: u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u>
Jan 26 03:39:26 monitor systemd[1]: grafana-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 26 03:39:26 monitor systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jan 26 03:39:26 monitor systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
Jan 26 03:39:26 monitor systemd[1]: grafana-server.service: Start request repeated too quickly.
Jan 26 03:39:26 monitor systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jan 26 03:39:26 monitor systemd[1]: Failed to start Grafana instance.              

So it seems that /etc/grafana/grafana.ini was broken.
According to the log, the server started again successfully at 09:01:16


Related issues 2 (2 open0 closed)

Related to openQA Infrastructure (public) - action #175710: OSD openqa.ini is corrupted, invalid characters, again 2025-01-17Blockedokurz2024-07-10

Actions
Blocked by openQA Infrastructure (public) - action #176250: file corruption in salt controlled config files size:MWorkable

Actions
Actions

Also available in: Atom PDF