action #107917
Updated by mkittler almost 3 years ago
### Observation
The corresponding GitLab pipeline failed: `monitor-o3 | Failed pipeline for master | fee77e0e`
```
$ ssh o3 'ping -q -c 1 imagetester >/dev/null' || ipmitool -I lanplus -C 3 -H 10.160.65.195 -U ADMIN -P $imagetester_ipmi_password power cycle
Error: Unable to establish IPMI v2 / RMCP+ session
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: command terminated with exit code 1
```
I haven't restarted the job because imagetester seems to be online nevertheless. IPMI being sometimes unavailable is something I also experience when using it manually. We could implement a retry, though.
## Suggestions
* Check if imagester is currently actually online or needs recovery
* Maybe the ping fails but the machine is online?
* Crosscheck credentials and IPMI access
* Re-try ipmi if it fails
* Check our wiki because we stated that imagetester does not have a working IPMI anyway
* The recovery is implemented in https://gitlab.suse.de/openqa/monitor-o3/-/blob/master/.gitlab-ci.yml
* imagetester has only been added recently: https://gitlab.suse.de/openqa/monitor-o3/-/commit/0a3d0ee4f9543961f8bb368ece1fcf3642b2a6bc