Project

General

Profile

action #157441

Updated by okurz 9 months ago

## Observation 

 https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2398227 
 ``` 
 Date: Sun, 17 Mar 2024 05:49:14 +0000 
 Date: Mon, 18 Mar 2024 05:49:42 +0000 

 qesapworker-prg5.qa.suse.cz: 
 2184      Minion did not return. [Not connected] 
 ``` 


 https://stats.openqa-monitor.qa.suse.de/alerting/grafana/host_up_alert_qesapworker-prg5/view?orgId=1 

 The worker seemed to have hung up. No login prompt on serial tty. 
 Rebooted via IPMI. 
 Worker came up, but a systemd service failed: … 

 It seems like the NVMe disk is not found anymore. Maybe it died and the system subsequently freezed. 

 ## Rollback steps 
 * ssh osd `sudo salt-key -y -a qesapworker-prg5.qa.suse.cz`

Back