QA » openQA Project » openQA Infrastructure

estimated with nsinger

arm-1
15:13: https://gitlab.suse.de/openqa/grafana-webhook-actions/-/pipelines/660066 => Recovery over IPMI (succeeded)
15:53: https://gitlab.suse.de/openqa/grafana-webhook-actions/-/pipelines/660172 => 3 PDU retries, failed to complete but pipeline succeeded

arm-2
15:53: https://gitlab.suse.de/openqa/grafana-webhook-actions/-/jobs/1531713 => 3 PDU retries, failed to complete but pipeline succeeded

arm-3
15:53: https://gitlab.suse.de/openqa/grafana-webhook-actions/-/pipelines/660173 => PDU login failed, no retry and pipeline failed

Our current configuration doesn't supply a infra ticket e-mail but rather falls back to the default osd-admins (https://gitlab.suse.de/openqa/grafana-webhook-actions/-/blob/master/ipmi-recover-worker#L22). We got these two messages as expected for arm-1 and arm-2.
What stands out to me are the "command cancelled" mails we got from the PDU. I can only guess to explain these by the current implementation of the expect-script with a "Immediate Reboot" followed by a "Immediate On". A "reboot" has a delay of 5 seconds (e.g. https://gitlab.suse.de/openqa/grafana-webhook-actions/-/jobs/1531709#L96) but the expect-script doesn't wait so it might happen that the "Immediate On" happens inside this "sleep period" of the PDU which causes a cancellation of the previous reboot sequence.

The second observation is a failed recovery attempt for arm-3. Here expect was unable to login into the PDU and got no response via telnet. I saw this behavior previously if a session is already opened. Given that we had 3 pipelines running at the exact same time all connecting to qaps06nue I think we just got unlucky and two expect-scripts where running at the same time trying to login into the PDU.

Suggestions:

1. Adjust expect-script to either use "power off" followed by a "power on" or only "reboot"
2. Adjust recovery script to allow expect to fail without the pipeline failing: https://gitlab.suse.de/openqa/grafana-webhook-actions/-/blob/master/ipmi-recover-worker#L71
3. Ensure only one pipeline accesses our PDU 3.1 Make pipelines run in sequence not parallel 3.2 Use different method to issue commands to the PDU. I had great success with SNMP previously.

Setting due date based on mean cycle time of SUSE QE Tools

nicksinger wrote:

1. check if the alert triggering mechanism via separate alert-channels still works as expected or if a single machine causes all three pipelines to run

Pipeline/Alertingflow is still working as expected after the migration. There is a separate "contact point" per worker in https://stats.openqa-monitor.qa.suse.de/alerting/notifications?alertmanager=grafana and https://gitlab.suse.de/openqa/grafana-webhook-actions/-/pipelines shows single pipelines for e.g. arm-3 and arm-2 on different timestamps (so no consecutive firing as feared). I think after https://gitlab.suse.de/openqa/grafana-webhook-actions/-/merge_requests/29 gets merged we can close this ticket.

https://gitlab.suse.de/openqa/grafana-webhook-actions/-/merge_requests/29 merged. I agree that we can close right away.