Project

General

Profile

Actions

action #131309

closed

[alert] NFS mount can fail due to hostname resolution error size:M

Added by okurz about 1 year ago. Updated 12 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2023-06-19
Due date:
2023-08-11
% Done:

0%

Estimated time:
Tags:

Description

Observations

martchus@worker11:~> sudo journalctl --since '2 days ago' -u var-lib-openqa-share.mount 
Jun 18 03:30:00 worker11 systemd[1]: Unmounting /var/lib/openqa/share...
Jun 18 03:30:00 worker11 systemd[1]: var-lib-openqa-share.mount: Deactivated successfully.
Jun 18 03:30:00 worker11 systemd[1]: Unmounted /var/lib/openqa/share.
-- Boot e08a8d421816483eb387db9aac85fdc4 --
Jun 18 03:36:02 worker11 systemd[1]: Mounting /var/lib/openqa/share...
Jun 18 03:36:02 worker11 mount[3411]: mount.nfs: Failed to resolve server openqa.suse.de: Name or service not known
Jun 18 03:36:02 worker11 systemd[1]: var-lib-openqa-share.mount: Mount process exited, code=exited, status=32/n/a
Jun 18 03:36:02 worker11 systemd[1]: var-lib-openqa-share.mount: Failed with result 'exit-code'.
Jun 18 03:36:02 worker11 systemd[1]: Failed to mount /var/lib/openqa/share.
Jun 18 03:37:03 worker11 systemd[1]: Mounting /var/lib/openqa/share...
Jun 18 03:37:05 worker11 mount[3803]: Created symlink /run/systemd/system/remote-fs.target.wants/rpc-statd.service → /usr/lib/systemd/system/rpc-statd.service.
Jun 18 03:37:06 worker11 systemd[1]: Mounted /var/lib/openqa/share.

the time shows that the machine worker11 rebooted during the planned weekly reboot window and likely OSD was also rebooting during that time and not yet immediately available again

Suggestions

  • worker11 is one of our two staging test machines. Crosscheck that it actually really has the latest salt state. For that power on both machines and bring them back as salt minions connected to OSD and ensure the salt high state is applied here
  • Crosscheck our mount options that we have configured in salt and locally in /etc/fstab, they should have some custom tweaking like systemd mount timeout, retry, etc.
  • Ensure that machines retry sufficiently often to mount from OSD to cover for periods when OSD itself is not up, e.g. as happened above in the usual weekly maintenance reboot window
  • This could be tested by mounting a share from one machine and keeping the server off for some minutes while the client reboots, e.g. power on worker11+12, create a NFS server on w11, mount it from w12, power off w11, reboot w12, wait and monitor how w12 behaves, then power on w11 and ensure that w12 eventually mounts and reaches the final systemd target without any failed services

Related issues 3 (1 open2 closed)

Related to openQA Infrastructure - action #158041: grenache needs upgrade to 15.5Resolvedokurz2024-03-262024-04-09

Actions
Related to openQA Infrastructure - action #163097: Share mount not working on openqaworker-arm-1 and other workers size:MResolvedmkittler2024-07-17

Actions
Copied to openQA Infrastructure - action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker servicesNew

Actions
Actions #2

Updated by nicksinger about 1 year ago

  • Assignee set to nicksinger
Actions #3

Updated by nicksinger 12 months ago

  • Status changed from Workable to In Progress

The defined options in https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/worker.sls#L86 are present on both workers. To not schedule tests I refrain for now to add them back into salt as it wouldn't bring any updates in that regard.

Going to check the last suggestion now to cross-mount between the workers and reboot them so see how it behaves.

Actions #4

Updated by nicksinger 12 months ago

Keeping worker12 shut-off while rebooting worker11. It behaved as we expected and did not fail. After starting worker12 I forgot to enable the nfs-server and therefore it timed out on accessing the share but recovered gracefully once I enabled the nfs-server process on worker12.
"Name or service not known" gives me the impression this has to do with DNS and a race with the network connection on boot. Checking if there are additional options or dependencies we can introduce.

Actions #5

Updated by okurz 12 months ago

Sounds very similar to the case we had when the openQA cache service failing to resolve DNS after bootup but worked fine when restarting. According to mkittler though we resolved that problem by using the IP address for localhost instead as we did not need a remote host connection here. Maybe something with glibc, getent, whatever, restarting nscd or simply retrying helps here?

Slightly related as we discussed: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/924 and https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/925

Actions #6

Updated by openqa_review 12 months ago

  • Due date set to 2023-08-11

Setting due date based on mean cycle time of SUSE QE Tools

Actions #7

Updated by nicksinger 12 months ago

While debugging in poo# I saw the following line from systemd: [104432.144929][ T1512] systemd-fstab-generator[1512]: x-systemd.device-timeout ignored for openqa.suse.de:/var/lib/openqa/share. Maybe this is related?

Actions #8

Updated by okurz 12 months ago

Good point. Maybe we never wrote the correct name for the setting or we use something that is not anymore supported or even not yet supported

Actions #9

Updated by nicksinger 12 months ago

  • Status changed from In Progress to Feedback

Couldn't find any conclusive results on why this option isn't supported but it shouldn't matter because we retry several times anyway.
Discovered an answer in https://suse.slack.com/archives/C02D92APKNU/p1690458456152179 and came up with a workaround to dynamically populate /etc/hosts based on DNS via salt: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/931
Given the several options we already tried I think this is the best we can do ATM

Actions #10

Updated by okurz 12 months ago

I agree, merged

Actions #11

Updated by nicksinger 12 months ago

  • Status changed from Feedback to Resolved

There was another fixup needed which was done in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/932
Multiple reboots showed no problems and we should be good again.

Actions #12

Updated by okurz 10 months ago

  • Copied to action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker services added
Actions #13

Updated by okurz 4 months ago

Actions #14

Updated by okurz 21 days ago

  • Related to action #163097: Share mount not working on openqaworker-arm-1 and other workers size:M added
Actions

Also available in: Atom PDF