action #80734
Updated by livdywan over 3 years ago
## Observation The [salt-states-openqa pipeline](https://gitlab.suse.de/openqa/salt-states-openqa/-/pipelines) is not always triggered even though [Automatic actions on Grafana](https://stats.openqa-monitor.qa.suse.de/d/1bNU0StZz/automatic-actions?orgId=1) shows a host to be offline, as happened *today* with `openqaworker-arm-1` (and which was meanwhile solved by triggering a reboot manually). A look at the logfile on `openqa-monitor.qa.suse.de` via `sudo cat /var/log/grafana/grafana.log | grep arm-1 | less` reveals: ``` t=2020-12-04T10:20:29+0100 lvl=eror msg="Failed to send webhook" logger=alerting.notifier.webhook error="Post \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=...&ref=master&variables[MACHINE]=openqaworker-arm-1\": \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=2fb7eb7083ff20058206746e50349b&ref=master&variables[MACHINE]=openqaworker-arm-1\": net/http: TLS handshake timeout" webhook="Trigger reboot of openqaworker-arm-1" t=2020-12-04T10:20:29+0100 lvl=eror msg="failed to send notification" logger=alerting.notifier uid=o5EYinpZk error="Post \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=...&ref=master&variables[MACHINE]=openqaworker-arm-1\": \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=2fb7eb7083ff20058206746e50349b&ref=master&variables[MACHINE]=openqaworker-arm-1\": net/http: TLS handshake timeout" t=2020-12-04T10:20:29+0100 lvl=eror msg="failed to send notification" logger=alerting.notifier uid=o5EYinpZk error="Post \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=...&ref=master&variables[MACHINE]=openqaworker-arm-1\": \"https://gitlab.suse.de/api/v4/projects/4652/trigger/pipeline?token=2fb7eb7083ff20058206746e50349b&ref=master&variables[MACHINE]=openqaworker-arm-1\": net/http: TLS handshake timeout" ``` So it looks like Grafana tried to post the API request, but failed due to TLS errors.