Project

General

Profile

action #132383

Updated by okurz over 1 year ago

## Observation 
 Multiple FC Basement OSD hosts are not reachable since 2023-07-06 01:50 CEST as visible on https://monitor.qa.suse.de/alerting/list?search=state:firing mentioning 
 * [openqa-piworker](https://monitor.qa.suse.de/alerting/grafana/host_up_alert_openqa-piworker/view?returnTo=%2Falerting%2Flist%3Fsearch%3Dstate%3Afiring) 
 * [openqaworker1](https://monitor.qa.suse.de/alerting/grafana/host_up_alert_openqaworker1/view?returnTo=%2Falerting%2Flist%3Fsearch%3Dstate%3Afiring) 
 * [sapworker2](https://monitor.qa.suse.de/alerting/grafana/host_up_alert_sapworker2/view?returnTo=%2Falerting%2Flist%3Fsearch%3Dstate%3Afiring) 
 * [sapworker3](https://monitor.qa.suse.de/alerting/grafana/host_up_alert_sapworker3/view?returnTo=%2Falerting%2Flist%3Fsearch%3Dstate%3Afiring) 

 plus the ["Packet loss" alert](https://monitor.qa.suse.de/alerting/grafana/2Z025iB4km/view?returnTo=%2Falerting%2Flist%3Fsearch%3Dstate%3Afiring) 

 ## Suggestions 
 * Wait for https://sd.suse.com/servicedesk/customer/portal/1/SD-126191 resolution 
 * Ensure all relevant machines in FC Basement are reachable again 
 * Retry https://gitlab.suse.de/openqa/osd-deployment/-/pipelines and other gitlab CI pipelines where failed 
 * Ensure alerts are ok again

Back