action #132383: FC Basement OSD hosts not reachable since 2023-07-06 01:50 CEST - openQA Infrastructure - openSUSE Project Management Tool

Actions

Copy link

action #132383

closed

FC Basement OSD hosts not reachable since 2023-07-06 01:50 CEST

Added by okurz 10 months ago. Updated 10 months ago.

Status:

Resolved

Priority:

Urgent

Assignee:

okurz

Category:

Target version:

openQA Project - Ready

Start date:

2023-07-06

Due date:

% Done:

Estimated time:

Tags:

alert, FC Basement, network outage

Description

Observation¶

Multiple FC Basement OSD hosts are not reachable since 2023-07-06 01:50 CEST as visible on https://monitor.qa.suse.de/alerting/list?search=state:firing mentioning

plus the "Packet loss" alert

Suggestions¶

Wait for https://sd.suse.com/servicedesk/customer/portal/1/SD-126191 resolution
Ensure all relevant machines in FC Basement are reachable again
Retry https://gitlab.suse.de/openqa/osd-deployment/-/pipelines and other gitlab CI pipelines where failed
Ensure alerts are ok again

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by okurz 10 months ago

Status changed from New to Blocked

https://sd.suse.com/servicedesk/customer/portal/1/SD-126191

Actions

Copy link

Updated by okurz 10 months ago

Description updated (diff)

Actions

Copy link

Updated by mkittler 10 months ago

It looks like that caused the NFS mount to be totally unresponsive. So the workers got stuck trying to access it during initialization (and thus stayed offline). Manual filesystem commands also get stuck. I could not even unmount the filesystem again. It is exactly the same on all 3 sap workers. I'll have a look on the other machines to see whether they are equally badly affected.

Actions

Copy link

Updated by mkittler 10 months ago

The other hosts (piworker and openqaworker1) were not affected. That's strange but also good.

The NFS mount being stuck was also the reason why zypper was stuck on rpm --root / --dbpath /usr/lib/sysimage/rpm -U --percent --noglob --force --nodeps -- /var/cache/zypp/packages/devel_openQA/x86_64/openQA-common-4.6.1688565452.efc15ea-lp155.5933.1.x86_64.rpm and thus the zypper lock was stuck as well and thus salt was not able to apply states. It would be nice if this chain of problems leading to other problems was at least shorter…

Actions

Copy link

Updated by okurz 10 months ago

Status changed from Blocked to In Progress

SD ticket was resolved, so network is back. Regarding the NFS mount: We tried to fade out the use of that for years for good reasons. I think we should try again, e.g. only provide the mount on some limited workers with a special worker class e.g. "deprecated-nfs".

Actions

Copy link