QA » openQA Project » openQA Infrastructure

Deleted salt key for openqaworker-arm-3 with sudo salt-key -d openqaworker-arm-3.suse.de and retriggered osd deployment pipeline on https://gitlab.suse.de/openqa/osd-deployment/pipelines . Running as https://gitlab.suse.de/openqa/osd-deployment/-/pipelines/106382

sol, power reset and power cycle don't work so it is not a surprise that our automatic recovery didn't work either. Maybe I can play around with IPMI a little bit more. Otherwise I suppose we'll have to resort to filing an infra ticket.

Skip to the end if you're not interested in my exploratory use of ipmitool :-)

ipmitool -I lanplus -C 3 -H openqaworker-arm-3-ipmi.suse.de -U ADMIN -P ADMIN chassis selftest
Self Test Results    : passed

ipmitool -I lanplus -C 3 -H openqaworker-arm-3-ipmi.suse.de -U ADMIN -P ADMIN chassis restart_cause
System restart cause: chassis power control command

ipmitool -I lanplus -C 3 -H openqaworker-arm-3-ipmi.suse.de -U ADMIN -P ADMIN chassis policy always-on
Set chassis power restore policy to always-on
ipmitool -I lanplus -C 3 -H openqaworker-arm-3-ipmi.suse.de -U ADMIN -P ADMIN chassis power reset
Set Chassis Power Control to Reset failed: Command not supported in present state
ipmitool -I lanplus -C 3 -H openqaworker-arm-3-ipmi.suse.de -U ADMIN -P ADMIN chassis power cycle
Set Chassis Power Control to Cycle failed: Command not supported in present state
ipmitool -I lanplus -C 3 -H openqaworker-arm-3-ipmi.suse.de -U ADMIN -P ADMIN chassis power diag
Set Chassis Power Control to Diag failed: Command not supported in present state

ipmitool -I lanplus -C 3 -H openqaworker-arm-3-ipmi.suse.de -U ADMIN -P ADMIN chassis power on
Chassis Power Control: Up/On
ipmitool -I lanplus -C 3 -H openqaworker-arm-3-ipmi.suse.de -U ADMIN -P ADMIN chassis power diag
Chassis Power Control: Diag

And something finally shows up on the SOL. It took a while as it conducted a few tests but now the system is booting. It looks like the chassis was turned off (whatever that means). Maybe our automatic recovery should also ensure that the chassis is turned on.

Wow, that's interesting. Can we just add a chassis on in the grafana-webhook actions repo?

If "grafana-webhook" is the place where we do the automatic recovery, than that's what I'd do. Still have to find it, though.

The SR has been merged and I've tested whether it works. It does. After power off the Grafana alert triggers the GitLab pipeline which then turns the power on again. However, since the power state is apparently not immediately reflected the subsequent power cycle command doesn't work:

PING openqaworker-arm-3.suse.de (10.160.0.85) 56(84) bytes of data.
--- openqaworker-arm-3.suse.de ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
Chassis Power Control: Up/On
Set Chassis Power Control to Cycle failed: Command not supported in present state

Not a big deal because it isn't required anyways in that state. If the power was on and the worker just otherwise stuck the power on command will be a noop which zero return code so this case should be fine as well.

discussed in weekly 2021-04-09:
https://gitlab.suse.de/openqa/grafana-webhook-actions/-/blob/master/ipmi-health-check#L22 wastes 10 minutes and also we found out that https://gitlab.suse.de/openqa/grafana-webhook-actions/-/merge_requests/12 did more than what we expected. I read all comments from #76876 again but have not found any discussion that we want to turn the "do-reboot" steps into a "check-health" script. So mkittler and me decided that instead we want to go back to the previous way to trigger a reboot as soon as the script is started, not check if the machine might be still up (respond to ping but potentially not being fully reactive) and not waste 10 minutes of waiting if ping comes back successful before triggering any action.

Also I suggest:

if $ipmitool power status | grep -q 'is on'; then
    $ipmitool power cycle
else
    $ipmitool power on
fi

SR for improving the problem mentioned in the previous comment: https://gitlab.suse.de/openqa/grafana-webhook-actions/-/merge_requests/14
for AC2: https://gitlab.suse.de/openqa/grafana-webhook-actions/-/merge_requests/15