Project

General

Profile

Actions
Copy link

action #166523

open

worker33 log entry "Couldn't write '1' to 'net/ipv4/conf/br1/forwarding', ignoring: No such file or directory" size:S

Added by okurz 6 months ago. Updated 6 days ago.

Status:
Workable
Priority:
Low
Assignee:
-
Category:
Regressions/Crashes
Target version:
QA (public) - future
Start date:
2024-09-09
Due date:
% Done:

0%

Estimated time:
Tags:
network, infra, reactive work

Description

Observation

While looking in the logs on worker I found

Sep 08 01:34:22 worker33 systemd-sysctl[1737]: Couldn't write '1' to 'net/ipv4/conf/br1/forwarding', ignoring: No such file or directory
Sep 08 01:34:22 worker33 systemd-sysctl[1737]: Couldn't write '1' to 'net/ipv4/conf/eth0/forwarding', ignoring: No such file or directory

Could be an incomplete path that should start with / or a race condition if the according prerequisities during a boot are not yet fulfilled. This seems to be related to https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch.sls?ref_type=heads#L41 where we write to the forwarding sysctl parameters.

Acceptance criteria

  • AC1: No error message like the above on all our salt controlled machines
  • AC2: openQA jobs on the affected machines can still reach the internet

Suggestions

  • Investigate the impact.
    • Maybe this affects multi-machine tests - which are currently limited
  • Check dependencies between systemd-sysctl (which sets this attribute) and wicked (or whichever service creates br1)
  • Lookup why this is a warning and not a failure of the service (or any service)
    • Read systemd-sysctl-related documentation
Actions
Copy link
 #1

Updated by okurz 6 months ago

  • Subject changed from worker33 log entry"Couldn't write '1' to 'net/ipv4/conf/br1/forwarding', ignoring: No such file or directory" to worker33 log entry "Couldn't write '1' to 'net/ipv4/conf/br1/forwarding', ignoring: No such file or directory" size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #2

Updated by okurz 8 days ago

  • Target version changed from Tools - Next to Ready
Actions
Copy link
 #3

Updated by mkittler 7 days ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions
Copy link
 #4

Updated by mkittler 7 days ago

Could be an incomplete path that should start with / or a race condition if the according prerequisites during a boot are not yet fulfilled.

The config file /etc/sysctl.d/99-salt.conf is written as expected by salt. Considering https://bbs.archlinux.org/viewtopic.php?id=285711 this is indeed a race condition and it is normal that the error message doesn't start with /. There are also more findings, e.g.:

So this is definitely a common problem and probably not related to Salt (which merely creates the config file).

Actions
Copy link
 #5

Updated by mkittler 7 days ago

The documentation says:

The network interface-specific options will also be applied individually for each network interface as it shows up in the system. (More specifically, net.ipv4.conf., net.ipv6.conf., net.ipv4.neigh.* and net.ipv6.neigh.*).

This would explain why this is just a warning: It'll try to apply the options later when the network interface shows up.

This would mean there's no impact and we can simply ignore those log message (except that the AC says otherwise). I also checked on worker33 and all those settings are applied. So far only the previously mentioned search results suggest that there might actually a problem.

Actions
Copy link
 #6

Updated by mkittler 7 days ago

Although the documentation also says:

This means that systemd-sysctl.service(8) which runs during early boot will not configure such parameters if they become available after it has run. To set such parameters, it is recommended to add an udev(7) rule to set those parameters when they become available.

And then it gives examples on how to do that. I'm wondering whether we still need this at all, though. The config was introduced in https://gitlab.suse.de/openqa/salt-states-openqa/-/commit/523cdef4.

Actions
Copy link
 #7

Updated by openqa_review 6 days ago

  • Due date set to 2025-03-12

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #9

Updated by mkittler 6 days ago ยท Edited

  • Due date deleted (2025-03-12)
  • Status changed from In Progress to Workable
  • Assignee deleted (mkittler)
  • Target version changed from Ready to future

The the setup script we do the same as in the Salt recipe. So we might see the same log messages on o3 workers as well.

We discussed that this is not important right now. The setting seems to be applied at some point (see my previous comment) and even if it isn't we would probably not have a problem because we schedule clusters only on one host anymore.

Actions
Copy link
 #10

Updated by okurz 6 days ago

  • Priority changed from Normal to Low
Actions
Copy link

Also available in: Atom PDF