action #157726
Updated by okurz about 2 months ago
## Observation
https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2415705
```
worker37.oqa.prg2.suse.org:
Minion did not return. [Not connected]
worker36.oqa.prg2.suse.org:
Minion did not return. [Not connected]
worker38.oqa.prg2.suse.org:
Minion did not return. [Not connected]
worker39.oqa.prg2.suse.org:
Minion did not return. [Not connected]
```
## Acceptance criteria
* **AC1:** osd-deployment passes again
## Suggestions
* *DONE* Take machine out of production: https://progress.opensuse.org/projects/openqav3/wiki/#Take-machines-out-of-salt-controlled-production
* *DONE Remove machine XYZ from production `ssh osd "sudo salt-key -y -d XYZ"`
* Retrigger failed osd deployment CI pipeline
* Confirm if this is one or multiple, possibly already known issues
* Fix any potential hardware issue, e.g. with hardware replacement
* Ensure machines are back in production
## Rollback steps
* https://progress.opensuse.org/projects/openqav3/wiki/#Bring-back-machines-into-salt-controlled-production `for i in 36 37 38 39 ; do sudo `hostname=XYZ ssh osd "sudo salt-key -y -a worker$i.oqa.prg2.suse.org; done $hostname && sleep 30 && for i in 36 37 38 39 ; do sudo salt --state-output=changes "worker$i*" state.apply` $hostname state.apply"`