action #121282
closed
Recover storage.qa.suse.de size:S
Added by mkittler over 1 year ago.
Updated over 1 year ago.
Description
Observation¶
The server cannot be reached via SSH and the host-up alert in our monitoring fired (paused it for now). I wasn't able to reach the IPMI host (via jumpy@qe-jumpy.suse.de as documented in pillars).
Acceptance criteria¶
- AC1: Machine is racked again
- AC2: Racktables is updated including mac/connections/rack
- AC3: No related alerts for that machine is firing
Rollback steps¶
- Tags changed from alert to alert, reactive work
- Related to action #88546: Make use of the new "Storage Server", e.g. complete OSD backup added
- Related to action #120267: Conduct the migration of openqa-ses aka. "storage.qa.suse.de" size:M added
I suppose having it in FC will be fine. That means we'd likely mount it somewhere in SRV2 or the lab in the 2nd floor until FC is ready.
Where's the machine now, btw?
Next time we could at least check what services run on a machine before pulling the plug. In this case it would have been very obvious that it's just the storage server.
- Tags changed from alert, reactive work to alert, next-office-day
- Subject changed from Recover storage.qa.suse.de to Recover storage.qa.suse.de size:S
- Description updated (diff)
- Status changed from New to Workable
- Related to action #69577: Handle installation of the new "Storage Server" added
- Description updated (diff)
At next opportunity someone nearby to Nuremberg please create a EngInfra ticket over sd.suse.com/ with added Jira SD group "OSD Admins", make an appointment to take the machine from 2.2.14 (TAM) QA lab location where the machine is currently located, bring it back to SRV1 and mount it back where it was, update racktables and make sure the machine is reachable. Migrating the machine into the new network zone should be done in #120267
As discussed in daily 2022-12-07 dheidler will pick this up.
- Status changed from Workable to Blocked
- Priority changed from Urgent to Normal
In the ticket gschlotter informed that they likely won't be at Maxtorhof anymore this year so we will have to wait
- Tags changed from alert, next-office-day, infra to alert, next-office-day, infra, reactive work
- Status changed from Blocked to In Progress
We plugged the machine back in where previously power8 was sitting. Gerhard configured the switch-ports to be in VLAN2 again. IPMI+OS is running and pingable again, racktables is updated
alerts enabled again. Checking in 5m if they come up green
- Status changed from In Progress to Resolved
- Assignee changed from dheidler to nicksinger
Great work on resolving this and taking care about all the mentioned alerts, appreciated. Now we can look into the previously blocked #120267
- Related to action #123082: backup of o3 to storage.qa.suse.de was not conducted by rsnapshot since 2021-12 size:M added
Also available in: Atom
PDF