action #175707
Updated by okurz 3 months ago
## Observation
OSD webUI ended up with "502 Bad Gateway" from nginx on 2025-01-17, needed manual restart of openqa-webui with `systemctl restart openqa-webui` by okurz.
```
# ls /home/rsnapshot/*/
/home/rsnapshot/alpha.0/:
jenkins.qa.suse.de localhost openqa.opensuse.org
/home/rsnapshot/alpha.1/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/alpha.2/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/alpha.3/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/alpha.4/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/alpha.5/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/beta.0/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/beta.1/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/beta.2/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/beta.3/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/beta.4/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/beta.5/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/beta.6/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/_delete.17191/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/_delete.17511/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org openqa.suse.de s.qa.suse.de
/home/rsnapshot/_delete.4193/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/delta.0/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org openqa.suse.de s.qa.suse.de
/home/rsnapshot/delta.1/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org openqa.suse.de s.qa.suse.de
/home/rsnapshot/delta.2/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org openqa.suse.de s.qa.suse.de
/home/rsnapshot/gamma.0/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/gamma.1/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/gamma.2/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
/home/rsnapshot/gamma.3/:
jenkins.qa.suse.de localhost openqa-monitor.qa.suse.de openqa.opensuse.org s.qa.suse.de
```
observe that openqa.suse.de is missing in all alpha, beta, gamma snapshots
## Acceptance criteria
* **AC1:** We have alerts when backups are missing
* **AC2:** There are again current backups from OSD on backup-vm
## Suggestions
* Observe that openqa.suse.de is missing in all alpha, beta, gamma snapshots. Possibly related to CC-related firewall changes preventing direct ssh access.
* Also be aware of related #173674
* Config for rsnapshot is in https://gitlab.suse.de/qa-sle/backup-server-salt
* Consider also alerting
Back