QA » openQA Project » openQA Infrastructure

Observation¶

The salt-states-openqa pipeline is not always triggered even though Automatic actions on Grafana shows a host to be offline, as happened today with openqaworker-arm-1 (and which was meanwhile solved by triggering a reboot manually).

A look at the logfile on openqa-monitor.qa.suse.de via sudo cat /var/log/grafana/grafana.log | grep arm-1 | less reveals:

t=2020-12-04T10:20:29+0100 lvl=eror msg="Failed to send webhook" logger=alerting.notifier.webhook error="Post \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=...&ref=master&variables[MACHINE]=openqaworker-arm-1\": net/http: TLS handshake timeout" webhook="Trigger reboot of openqaworker-arm-1"
t=2020-12-04T10:20:29+0100 lvl=eror msg="failed to send notification" logger=alerting.notifier uid=o5EYinpZk error="Post \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=...&ref=master&variables[MACHINE]=openqaworker-arm-1\": net/http: TLS handshake timeout"
t=2020-12-04T10:20:29+0100 lvl=eror msg="failed to send notification" logger=alerting.notifier uid=o5EYinpZk error="Post \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=...&ref=master&variables[MACHINE]=openqaworker-arm-1\": net/http: TLS handshake timeout"

So it looks like Grafana tried to post the API request, but failed due to TLS errors.

See also RT-ADM #181878