Project

General

Profile

Actions
Copy link

action #135509

closed

monitor.qa.suse.de yields 502 Bad Gateway from nginx/1.21.5 size:M

Added by okurz 8 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
nicksinger
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-09-11
Due date:
% Done:

0%

Estimated time:
Tags:
monitoring, grafana, infra

Description

Observation

monitor.qa.suse.de yields 502 Bad Gateway from nginx/1.21.5. We looked into logs of the service granafa-server already, couldn't start. Seems to have started just while we looked at it around 2023-09-11 11:00Z

Also https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1820468:

ID: grafana-server
1037    Function: service.running
1038      Result: False
1039     Comment: Job for grafana-server.service failed because the control process exited with error code.
1040              See "systemctl status grafana-server.service" and "journalctl -xeu grafana-server.service" for details.
1041     Started: 13:15:54.262049
1042    Duration: 16566.052 ms
1043     Changes:   
1044Summary for openqa-monitor.qa.suse.de

Steps to reproduce

Suggestions

  • On monitor.qa.suse.de look into journalctl -u grafana-server
  • Just try a plain reboot of the whole VM
  • Fix the problem
  • Consider improvements to fix in the future
Actions
Copy link
 #1

Updated by okurz 8 months ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by nicksinger 8 months ago

  • Status changed from In Progress to Feedback

The grafana-server-service was not able to access /var/run/grafana where grafanas unix socket is created. This was most likely caused because we manage this folder in salt. After a cleanup the folder was not present and salt was not running to create it again before grafana-server tried to start. I changed our states to create a according systemd-tmpfiles config entry: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/982

I think systemd-tmpfiles is also responsible for the previous cleanup.

Actions
Copy link
 #3

Updated by nicksinger 8 months ago

  • Status changed from Feedback to Resolved

change was deployed and I verified manually that systemd-tmpfiles --create is only executed on file changes

Actions
Copy link

Also available in: Atom PDF