QA » openQA Project » openQA Infrastructure

This is what it how an exclusion could be done (not tested yet): https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1218

mkittler wrote in #note-3:

The journal looks like this:

sudo journalctl -fu var-lib-openqa-share.automount
…
-- Boot 2e0ffe940b3e4639ae27151d93e4f9ef --
Jul 02 02:41:33 openqaworker-arm-1 systemd[1]: Set up automount var-lib-openqa-share.automount.
Jul 02 02:53:32 openqaworker-arm-1 systemd[1]: var-lib-openqa-share.automount: Got automount request for /var/lib/openqa/share, triggered by 2590 (worker)
Jul 02 03:01:08 openqaworker-arm-1 systemd[1]: var-lib-openqa-share.automount: Got automount request for /var/lib/openqa/share, triggered by 2590 (worker)
Jul 02 04:48:38 openqaworker-arm-1 systemd[1]: var-lib-openqa-share.automount: Got hangup/error on autofs pipe from kernel. Likely our automount point has been unmounted by someone or something else?
Jul 02 04:48:38 openqaworker-arm-1 systemd[1]: var-lib-openqa-share.automount: Failed with result 'unmounted'.

Judging by the logs this is a recurring problem. Unless someone restarted the unit manually on Jul 02 07:16:00 the problem fixed itself (which is not visible in the journal but on Grafana).

The only relevant log lines in dmesg are:

[  783.324257] RPC: Registered tcp NFSv4.1 backchannel transport module.
[  783.532483] Key type dns_resolver registered
[  783.958133] NFS: Registering the id_resolver key type

When looking at the last 30 days in Grafana it becomes very apparent that the var-lib-openqa-share.automount unit failed on various hosts on various occasions. So this is really not a new problem and not specific to arm-1.

Correct. I saw that problem the past weeks but failed to find corresponding reports in tickets where I would have commented about that.

Considering there apparently is a retry going on and the problem always fixes itself (or was ever manual invention required?)

Yes, AFAIR I needed to apply manual intervention.

I suggest we exclude this unit from the systemd services alert.

Hm, I am not convinced about that. Maybe we can find a pattern under which conditions this failed on the various hosts? And as there is a retry already included why do we end up with failed systemd units then?

By the way I see this related to https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/925 : If we would manage to not need to rely on the NFS share from OSD to workers then we wouldn't need the mount at all anymore preventing multiple problems.

Also worker40:

2024-07-02 14:58:00 worker40 var-lib-openqa-share.automount 1

The latest version of my MR for ignoring should work now, I tested it via:

sudo bash -c "salt-call --out=json \\
    --pillar-root=../salt-pillars-openqa --local slsutil.renderer \\
    '$PWD/monitoring/telegraf/telegraf-common.conf' \\
    default_renderer=jinja host=foo \\
  | jq -r '.local'"

I invoked systemctl reset-failed on worker40 because the unit was indeed failing there but the NFS mount was actually functional.

I had another look on arm-1 and there the NFS mount was actually not functional. So I started var-lib-openqa-share.automount there again which worked immediately.

I though it might make sense to also ensure var-lib-openqa-share.automount is started on worker40 but starting it failed via var-lib-openqa-share.automount: Path /var/lib/openqa/share is already a mount point, refusing start.. So maybe the mount on worker40 was restored by other means than the automount unit. I'll keep it as-is.

Considering this means that the NFS mount is not functional after all I'm not so sure anymore whether we should ignore if this unit fails. However, having to deal with this alert/problem so often manually is also not ideal.

Useful documentation:

• https://www.suse.com/support/kb/doc/?id=000020402
• https://man7.org/linux/man-pages/man5/systemd.automount.5.html

This is happening more often since 2024-06-14 07:49:00, see https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=1716816941347&to=1720002135377.

Three points that you could follow, indepentant of each other:

1. Implement a custom systemd restart unit and custom systemd check+monitoring unit and blocklist the .automount units
2. Research how a systemd automount unit which is not a service could be restarted on failure: For this I found an open feature request https://github.com/systemd/systemd/issues/16811 with workaround in https://github.com/systemd/systemd/issues/16811#issuecomment-728662590
3. Research about the error "Got hangup/error on autofs pipe from kernel. Likely our automount point has been unmounted by someone or something else?". As we haven't found anything in web search I suggest to create a bug on bugzilla.suse.com plus ask experts in SUSE internal chat as well as external upstream chat

I looked into those points from 3 to 1.

\3. This search isn't really revealing. It leads to https://github.com/systemd/systemd/blob/main/src/core/automount.c and of course a few forum posts and issues without clear resolution.
\2. This is an open issue https://github.com/systemd/systemd/issues/16811. It mentions how to workaround it similar to our own idea in point 1. There's also the closed issue https://github.com/systemd/systemd/issues/4468 about mount units themselves which doesn't read very promising as well.
\1. I'll go for that option using the code from https://github.com/systemd/systemd/issues/16811#issuecomment-728662590.