QA » openQA Project » openQA Infrastructure

Same issue as last time, machine did not automatically apply its network configuration after a reboot:

qa-jump:~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 72:1c:68:af:4d:17 brd ff:ff:ff:ff:ff:ff
    altname enp0s2
    altname ens2
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:3b:4c:8c brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3

qa-jump:~ # rcnetwork restart

qa-jump:~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 72:1c:68:af:4d:17 brd ff:ff:ff:ff:ff:ff
    altname enp0s2
    altname ens2
    inet 10.168.192.10/22 brd 10.168.195.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::701c:68ff:feaf:4d17/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:3b:4c:8c brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 192.168.44.1/24 brd 192.168.44.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe3b:4c8c/64 scope link
       valid_lft forever preferred_lft forever

qa-jump:~ # systemctl --failed
  UNIT             LOAD   ACTIVE SUB    DESCRIPTION
● mnt-dist.mount   loaded failed failed /mnt/dist
● mnt-openqa.mount loaded failed failed /mnt/openqa

qa-jump:~ # systemctl restart mnt-dist.mount
qa-jump:~ # systemctl restart mnt-openqa.mount

This seems ok:

qa-jump:~ # systemctl is-enabled network
alias
qa-jump:~ # systemctl is-enabled wicked
enabled

Not sure yet what's different with this machine.

There is an issue with the mounts, causing wicked to fail during boot:

Jun 28 10:25:09 qa-jump systemd[1]: mnt-openqa.mount: Found ordering cycle on network-online.target/start
Jun 28 10:25:09 qa-jump systemd[1]: mnt-openqa.mount: Found dependency on wicked.service/start
Jun 28 10:25:09 qa-jump systemd[1]: mnt-openqa.mount: Found dependency on wickedd-auto4.service/start
Jun 28 10:25:09 qa-jump systemd[1]: mnt-openqa.mount: Found dependency on local-fs.target/start
Jun 28 10:25:09 qa-jump systemd[1]: mnt-openqa.mount: Found dependency on srv-tftpboot-mnt-openqa.mount/start
Jun 28 10:25:09 qa-jump systemd[1]: mnt-openqa.mount: Found dependency on mnt-openqa.mount/start
Jun 28 10:25:09 qa-jump systemd[1]: mnt-openqa.mount: Job local-fs.target/start deleted to break ordering cycle starting with mnt-openqa.mount/start

This seems like a similar issue as the one described in https://www.suse.com/support/kb/doc/?id=000018796.

I edited the fstab entry from @okurz respectively by appending _netdev to its options:

# okurz: 2023-05-17: "journalctl -e" revealed that systems (still?) try to access /mounts/dist so adding that bind mount additionally for convenience but that should be streamlined
# gpfuetzenreuter: 2023-06-28: repair broken network dependency cycle by adding _netdev
/mnt/dist                                       /mounts/dist                  none     defaults,bind,_netdev    0 0

After another reboot, the network came up fine.

All affected jobs restarted:

{
  result   => [{ 11462148 => 11469067 }],
  test_url => [{ 11462148 => "/tests/11469067" }],
}
{
  result   => [{ 11468047 => 11469068 }],
  test_url => [{ 11468047 => "/tests/11469068" }],
}
{
  errors   => ["Specified job 11467991 has already been cloned as 11468908"],
  result   => [],
  test_url => [],
}
{
  errors   => ["Specified job 11462542 has already been cloned as 11467992"],
  result   => [],
  test_url => [],
}
{
  result   => [{ 11463220 => 11469082 }],
  test_url => [{ 11463220 => "/tests/11469082" }],
}
{
  errors   => ["Specified job 11463236 has already been cloned as 11468027"],
  result   => [],
  test_url => [],
}
{
  errors   => ["Specified job 11463194 has already been cloned as 11468046"],
  result   => [],
  test_url => [],
}
{
  errors   => ["Specified job 11463213 has already been cloned as 11468047"],
  result   => [],
  test_url => [],
}
{
  result   => [{ 11468046 => 11469088 }],
  test_url => [{ 11468046 => "/tests/11469088" }],
}
{
  errors   => ["Specified job 11463228 has already been cloned as 11467988"],
  result   => [],
  test_url => [],
}
{
  result   => [{ 11467826 => 11469089 }],
  test_url => [{ 11467826 => "/tests/11469089" }],
}
{
  errors   => ["Specified job 11462530 has already been cloned as 11467991"],
  result   => [],
  test_url => [],
}
{
  result   => [{ 11468045 => 11469090 }],
  test_url => [{ 11468045 => "/tests/11469090" }],
}
{
  result   => [{ 11449788 => 11469091 }],
  test_url => [{ 11449788 => "/tests/11469091" }],
}
{
  errors   => ["Specified job 11464106 has already been cloned as 11468016"],
  result   => [],
  test_url => [],
}
{
  errors   => ["Specified job 11462129 has already been cloned as 11468045"],
  result   => [],
  test_url => [],
}