Project

General

Profile

Actions
Copy link

action #75016

closed

[osd-admins][alert] Failed systemd services alert (workers): os-autoinst-openvswitch.service (and var-lib-openqa-share.mount) on openqaworker-arm-2 and others

Added by okurz over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
mkittler
Category:
-
Target version:
openQA Project - Ready
Start date:
2020-10-21
Due date:
% Done:

0%

Estimated time:
Tags:
arm, alert, osd

Description

Observation

```From: Grafana osd-admins@suse.de
To: osd-admins@suse.de
Sender: osd-admins
List-Id:
Date: 21/10/2020 09.06

/[OK] Failed systemd services alert (workers)/ 


http://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?fullscreen&edit&tab=alert&panelId=6&orgId=1

## Problem

I found that this is on openqaworker-arm-2, the service failed is "os-autoinst-openvswitch.service":

● os-autoinst-openvswitch.service - os-autoinst openvswitch helper
Loaded: loaded (/usr/lib/systemd/system/os-autoinst-openvswitch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/os-autoinst-openvswitch.service.d
└─override.conf
Active: failed (Result: timeout) since Wed 2020-10-21 07:41:11 UTC; 1h 56min ago

Oct 21 07:39:41 openqaworker-arm-2 systemd[1]: Starting os-autoinst openvswitch helper...
Oct 21 07:40:42 openqaworker-arm-2 wicked[3318]: device br1: unable to apply configuration to nanny
Oct 21 07:41:11 openqaworker-arm-2 systemd[1]: os-autoinst-openvswitch.service: Start-pre operation timed out. Terminating.
Oct 21 07:41:11 openqaworker-arm-2 systemd[1]: Failed to start os-autoinst openvswitch helper.
Oct 21 07:41:11 openqaworker-arm-2 systemd[1]: os-autoinst-openvswitch.service: Unit entered failed state.
Oct 21 07:41:11 openqaworker-arm-2 systemd[1]: os-autoinst-openvswitch.service: Failed with result 'timeout'.

Related issues 6 (0 open — 6 closed)

Related to openQA Infrastructure - action #73633: OSD partially unresponsive, triggering 500 responses, spotty response visible in monitoring panels but no alert triggered (yet)Resolvednicksinger2020-10-202020-11-17

Actions
Related to openQA Infrastructure - action #75238: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.2Resolvedlivdywan

Actions
Related to openQA Infrastructure - action #89551: NFS mount fails after boot (reproducible on some OSD workers)Resolvedmkittler2021-03-052021-03-31

Actions
Related to openQA Infrastructure - action #158041: grenache needs upgrade to 15.5Resolvedokurz2024-03-262024-04-09

Actions
Copied to openQA Infrastructure - action #75274: [osd-admins][alert][learning] Failed systemd services alert (workers): os-autoinst-openvswitch.service aborts retries after 60s and is not easily configurableResolvedlivdywan2020-12-04

Actions
Copied to openQA Infrastructure - action #75277: enable "postfix" service again after #73633 resolvedResolvedokurz

Actions
Actions
Copy link
 #1

Updated by okurz over 3 years ago

Manually restarted the service and the service turned out fine. As also there were currently no openQA jobs running I triggered a reboot of the machine to crosscheck if that reproduces the problem.

EDIT: After a reboot now not only os-autoinst-openvswitch.service fails again but also var-lib-openqa-share.mount . Might this be related to #73633 ?

It seems like the network has severe problems here as well in being very slow but still there in the end:

> sudo journalctl -b -u var-lib-openqa-share.mount
-- Logs begin at Thu 2019-12-05 13:55:56 UTC, end at Wed 2020-10-21 10:14:02 UTC. --
Oct 21 09:54:25 openqaworker-arm-2 systemd[1]: Mounting /var/lib/openqa/share...
Oct 21 09:54:26 openqaworker-arm-2 systemd[1]: var-lib-openqa-share.mount: Mount process exited, code=exited status=32
Oct 21 09:54:26 openqaworker-arm-2 systemd[1]: Failed to mount /var/lib/openqa/share.
Oct 21 09:54:26 openqaworker-arm-2 systemd[1]: var-lib-openqa-share.mount: Unit entered failed state.
Oct 21 09:54:31 openqaworker-arm-2 systemd[1]: Mounting /var/lib/openqa/share...
…
Oct 21 09:56:37 openqaworker-arm-2 systemd[1]: Mounting /var/lib/openqa/share...
Oct 21 09:56:37 openqaworker-arm-2 systemd[1]: var-lib-openqa-share.mount: Mount process exited, code=exited status=32
Oct 21 09:56:37 openqaworker-arm-2 systemd[1]: Failed to mount /var/lib/openqa/share.
Oct 21 09:56:37 openqaworker-arm-2 systemd[1]: var-lib-openqa-share.mount: Unit entered failed state.
Oct 21 10:13:56 openqaworker-arm-2 systemd[1]: Mounting /var/lib/openqa/share...
Oct 21 10:14:00 openqaworker-arm-2 systemd[1]: Mounted /var/lib/openqa/share.

also here a manual restart of the service helped. Same for
os-autoinst-openvswitch.service

Actions
Copy link
 #2

Updated by okurz over 3 years ago

  • Related to action #73633: OSD partially unresponsive, triggering 500 responses, spotty response visible in monitoring panels but no alert triggered (yet) added
Actions
Copy link
 #3

Updated by okurz over 3 years ago

  • Subject changed from [osd-admins][alert] Failed systemd services alert (workers) to [osd-admins][alert] Failed systemd services alert (workers): os-autoinst-openvswitch.service (and var-lib-openqa-share.mount) on openqaworker-arm-2
  • Status changed from In Progress to Blocked

I am suspecting that this problem is due to #73633 hence blocking by that

EDIT: As suggested by nsinger I could disable IPv6 completely and crosscheck with echo net.ipv6.conf.all.disable_ipv6 = 1 > /etc/sysctl.d/poo73633_debugging.conf. But first I am conducting the experiment to reboot multiple times and see if the machine fails to come up. For this I have paused the alerts "[openqa] openqaworker-arm-2 online (long-time) alert" and "openqaworker-arm-2 offline" and disabled the salt-minion and telegraf and openqa-services on openqaworker-arm-2:

sudo systemctl disable --now salt-minion telegraf openqa-worker.target openqa-worker-cacheservice openqa-worker-cacheservice-minion

Then rebooted (got stuck in initial BIOS, had to exit manually and continue), but then reproduced the problem in both services over multiple reboots. That makes 4/4 failures.

I found in journalctl searching for "(var-lib-openqa-share|IPv6)":

Oct 21 11:31:29 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
…
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: var-lib-openqa-share.mount: Mount process exited, code=exited status=32
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: Failed to mount /var/lib/openqa/share.
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: Dependency failed for Remote File Systems.
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: var-lib-openqa-share.mount: Unit entered failed state.
…
Oct 21 11:35:01 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

so it takes 90s before the service var-lib-openqa-share.mount is aborted and only 4m after boot there is IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready.

Disabled IPv6 now and triggered reboot. But that showed the same problem and additionally failed postfix. Did sudo rm /etc/sysctl.d/poo00000_debugging.conf. Comparing how long it takes for "eth1" to come up I checked with sudo journalctl --since="2020-10-14" | grep '\(Booting Linux\|eth1.*[lL]ink\)' looking for "eth1" I found:

sudo journalctl --since="2020-10-14" | grep '\(Booting Linux\|eth1.*[lL]ink\)'
Oct 14 09:22:50 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 14 09:23:53 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 14 09:23:53 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 18 01:36:37 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 18 01:37:41 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 18 01:37:41 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 18 23:36:52 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 18 23:37:51 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 18 23:37:51 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 18 23:38:53 openqaworker-arm-2 wickedd-dhcp6[2051]: eth1: link confirmation in reply with status All addresses still on link.
Oct 21 07:38:14 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 07:41:46 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 07:41:46 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 09:47:30 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 09:52:55 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 09:56:29 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 09:56:29 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 10:47:27 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 10:51:22 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 10:51:22 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 11:13:51 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 11:17:48 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 11:17:48 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 11:31:29 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 11:35:01 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 11:35:01 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 11:47:49 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 11:51:51 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex

so I see in the past, Oct 18 and before a time of around 60s from bootup to the time when the link becomes ready but no problem with IPv6 but a significant time increase starting Oct 21 (there was no reboot in between). Now it takes 3-8m.

EDIT: I ran the same lookup against all machines within OSD infrastructure. What we got:

okurz@openqa:~> sudo salt \* cmd.run "sudo journalctl --since='2020-10-14' | grep '\(kernel: Linux version\|eth.*Link is Up\|eth.*link becomes ready\)'"
malbec.arch.suse.de:
    Oct 18 03:35:11 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:36:39 malbec kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
openqaworker10.suse.de:
    Oct 18 03:33:05 openqaworker10 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:01 openqaworker10 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:01 openqaworker10 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
QA-Power8-4-kvm.qa.suse.de:
    Oct 18 03:33:55 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:20 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
openqaworker2.suse.de:
    Oct 18 03:34:29 linux-1nn1 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:35:18 linux-1nn1 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
openqaworker3.suse.de:
    Oct 18 03:34:37 linux-5h37 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 19 14:28:57 linux-5h37 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 19 14:30:14 linux-5h37 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 19 14:55:10 linux-5h37 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 19 14:56:29 linux-5h37 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
openqaworker13.suse.de:
    Oct 18 03:33:04 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:08 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:08 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
grenache-1.qa.suse.de:
    Oct 18 03:33:29 grenache-1 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
openqaworker8.suse.de:
    Oct 14 10:26:22 linux-fwcx kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 14 10:26:58 linux-fwcx kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 14 10:26:58 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:33:38 linux-fwcx kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:15 linux-fwcx kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:15 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker9.suse.de:
    Oct 18 03:33:38 linux-q6bp kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:14 linux-q6bp kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:14 linux-q6bp kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker-arm-1.suse.de:
    Oct 16 11:40:21 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 16 11:45:52 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 16 11:47:13 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 16 11:47:13 openqaworker-arm-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker6.suse.de:
    Oct 18 03:36:31 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:37:17 localhost kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:37:17 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker5.suse.de:
    Oct 18 03:36:33 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:37:20 localhost kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:37:20 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker-arm-3.suse.de:
    Oct 14 20:26:27 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 14 20:28:16 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 14 20:28:16 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 15 00:22:23 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 15 00:24:08 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 15 00:24:08 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 16 13:02:25 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 16 13:04:14 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 16 13:04:14 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 16 22:52:33 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 16 22:53:51 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 16 22:53:51 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 02:27:23 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 18 02:29:23 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 02:29:23 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:36:08 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:37:24 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 03:37:24 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:42:16 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:44:09 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 03:44:09 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker8.suse.de:
    Oct 14 10:26:54 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 14 10:26:58 linux-fwcx kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 14 10:26:58 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:34:12 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 03:34:15 linux-fwcx kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:15 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker9.suse.de:
    Oct 18 03:34:11 linux-q6bp kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 03:34:14 linux-q6bp kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:14 linux-q6bp kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker-arm-3.suse.de:
    Oct 14 20:26:27 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 14 20:28:14 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 14 20:28:16 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 14 20:28:16 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 15 00:22:23 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 15 00:24:07 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 15 00:24:08 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 15 00:24:08 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 16 13:02:25 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 16 13:04:12 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 16 13:04:14 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 16 13:04:14 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 16 22:52:33 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 16 22:53:49 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 16 22:53:51 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 16 22:53:51 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 02:27:23 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 18 02:29:21 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 02:29:23 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 02:29:23 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:36:08 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 18 03:37:22 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 03:37:24 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 03:37:24 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:42:16 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 18 03:44:07 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 03:44:09 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 03:44:09 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

seems like no other machine rebooted during that time but we see consistent times around 60s (with outliers <120s) between boot and "link up" and "IPv6 link ready".

Created specific ticket I found that the network link-up time has significantly increased on the machine openqaworker-arm-2 since 2020-10-20, likely same on other machines.

I conducted the experiment to reboot multiple times and see if the machine fails to come up with some network reliant services.

I found in journalctl searching for "(var-lib-openqa-share|IPv6)":

Oct 21 11:31:29 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
…
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: var-lib-openqa-share.mount: Mount process exited, code=exited status=32
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: Failed to mount /var/lib/openqa/share.
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: Dependency failed for Remote File Systems.
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
Oct 21 11:33:01 openqaworker-arm-2 systemd[1]: var-lib-openqa-share.mount: Unit entered failed state.
…
Oct 21 11:35:01 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

so it takes 90s before the service var-lib-openqa-share.mount is aborted and only 4m after boot there is IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready.

Disabled IPv6 now and triggered reboot. But that showed the same problem. Comparing how long it takes for "eth1" to come up I checked with sudo journalctl --since="2020-10-14" | grep '\(Booting Linux\|eth1.*[lL]ink\)' looking for "eth1" I found:

sudo journalctl --since="2020-10-14" | grep '\(Booting Linux\|eth1.*[lL]ink\)'
Oct 14 09:22:50 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 14 09:23:53 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 14 09:23:53 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 18 01:36:37 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 18 01:37:41 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 18 01:37:41 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 18 23:36:52 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 18 23:37:51 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 18 23:37:51 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 18 23:38:53 openqaworker-arm-2 wickedd-dhcp6[2051]: eth1: link confirmation in reply with status All addresses still on link.
Oct 21 07:38:14 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 07:41:46 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 07:41:46 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 09:47:30 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 09:52:55 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 09:56:29 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 09:56:29 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 10:47:27 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 10:51:22 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 10:51:22 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 11:13:51 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 11:17:48 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 11:17:48 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 11:31:29 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 11:35:01 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
Oct 21 11:35:01 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 21 11:47:49 openqaworker-arm-2 kernel: Booting Linux on physical CPU 0x0000000000 [0x431f0a11]
Oct 21 11:51:51 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex

so I see in the past, Oct 18 and before a time of around 60s from bootup to the time when the link becomes ready but no problem with IPv6 but a significant time increase starting Oct 21 (there was no reboot in between). Now it takes 3-8m.
I ran the same lookup against all machines within OSD infrastructure. What we got:

okurz@openqa:~> sudo salt \* cmd.run "sudo journalctl --since='2020-10-14' | grep '\(kernel: Linux version\|eth.*Link is Up\|eth.*link becomes ready\)'"
malbec.arch.suse.de:
    Oct 18 03:35:11 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:36:39 malbec kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
openqaworker10.suse.de:
    Oct 18 03:33:05 openqaworker10 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:01 openqaworker10 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:01 openqaworker10 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
QA-Power8-4-kvm.qa.suse.de:
    Oct 18 03:33:55 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:20 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
openqaworker2.suse.de:
    Oct 18 03:34:29 linux-1nn1 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:35:18 linux-1nn1 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
openqaworker3.suse.de:
    Oct 18 03:34:37 linux-5h37 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 19 14:28:57 linux-5h37 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 19 14:30:14 linux-5h37 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 19 14:55:10 linux-5h37 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 19 14:56:29 linux-5h37 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
openqaworker13.suse.de:
    Oct 18 03:33:04 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:08 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:08 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
grenache-1.qa.suse.de:
    Oct 18 03:33:29 grenache-1 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
openqaworker8.suse.de:
    Oct 14 10:26:22 linux-fwcx kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 14 10:26:58 linux-fwcx kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 14 10:26:58 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:33:38 linux-fwcx kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:15 linux-fwcx kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:15 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker9.suse.de:
    Oct 18 03:33:38 linux-q6bp kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:34:14 linux-q6bp kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:14 linux-q6bp kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker-arm-1.suse.de:
    Oct 16 11:40:21 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 16 11:45:52 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 16 11:47:13 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 16 11:47:13 openqaworker-arm-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker6.suse.de:
    Oct 18 03:36:31 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:37:17 localhost kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:37:17 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker5.suse.de:
    Oct 18 03:36:33 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:37:20 localhost kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:37:20 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker-arm-3.suse.de:
    Oct 14 20:26:27 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 14 20:28:16 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 14 20:28:16 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 15 00:22:23 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 15 00:24:08 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 15 00:24:08 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 16 13:02:25 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 16 13:04:14 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 16 13:04:14 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 16 22:52:33 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 16 22:53:51 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 16 22:53:51 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 02:27:23 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.71-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 8 12:40:48 UTC 2020 (dda1832)
    Oct 18 02:29:23 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 02:29:23 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:36:08 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:37:24 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 03:37:24 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:42:16 openqaworker-arm-3 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:44:09 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 03:44:09 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker8.suse.de:
    Oct 14 10:26:54 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 14 10:26:58 linux-fwcx kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 14 10:26:58 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:34:12 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 03:34:15 linux-fwcx kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:15 linux-fwcx kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker9.suse.de:
    Oct 18 03:34:11 linux-q6bp kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 03:34:14 linux-q6bp kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Oct 18 03:34:14 linux-q6bp kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker-arm-3.suse.de:
    Oct 14 20:26:27 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 14 20:28:14 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 14 20:28:16 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 14 20:28:16 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 15 00:22:23 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 15 00:24:07 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 15 00:24:08 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 15 00:24:08 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 16 13:02:25 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 16 13:04:12 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 16 13:04:14 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 16 13:04:14 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 16 22:52:33 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 16 22:53:49 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 16 22:53:51 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 16 22:53:51 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 02:27:23 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 18 02:29:21 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 02:29:23 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 02:29:23 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:36:08 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 18 03:37:22 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 03:37:24 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 03:37:24 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Oct 18 03:42:16 openqaworker-arm-3 kernel: Booting Linux on physical CPU 0x0
    Oct 18 03:44:07 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    Oct 18 03:44:09 openqaworker-arm-3 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Oct 18 03:44:09 openqaworker-arm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

seems like no other machine rebooted during that time but we see consistent times around 60s (with outliers <120s) between boot and "link up" and "IPv6 link ready".

Possibly related to https://infra.nue.suse.com/SelfService/Display.html?id=178694

Actions
Copy link
 #4

Updated by okurz over 3 years ago

The machine crashed again and caused failed systemd service alerts again. To prevent that I paused the alert for now and added retry on the systemd services:

sudo systemctl edit os-autoinst-openvswitch.service
[Service]
# https://progress.opensuse.org/issues/75016
Restart=on-failure
RestartSec=1m

and

sudo systemctl edit var-lib-openqa-share.mount
[Mount]
# https://progress.opensuse.org/issues/75016
TimeoutSec=600s

and rebooted. This should be removed again if the EngInfra problem was resolved.

EDIT: The retry on os-autoinst-openvswitch.service seems to work but not for var-lib-openqa-share.mount . In the meantime I have seen the same problem on other hosts, e.g. openqaworker-arm-1. Only other ideas for a better workaround would be to change the "Before=" and "After=" entries but I don't know to what.

Actions
Copy link
 #5

Updated by okurz over 3 years ago

Same problem happens on openqaworker8 now. Following #73633#note-8 I disabled IPv6 on openqaworker8 which causes postfix to fail so I temporarily disabled that service with systemctl disable --now postfix. However, also after reboot, var-lib-openqa-share.mount still fails.

After we fix the problems we should ensure to re-enable postfix again.

Now I tried on openqaworker8 for further workarounds and any timeout or retry changes in /etc/fstab to no avail. Researched further and found https://github.com/systemd/systemd/issues/4468 about a rejected systemd feature to have better retry, it has a lengthy discussion, grumpy opinions about systemd, a lot of up- and down-votes and is rejected. Haven't found a good suggestion for workaround. then I tried x-systemd.automount in /etc/fstab in the hope the mount point would only mount at a later point when tests would actually need it but somehow it still tried to mount immediately.

Actions
Copy link
 #6

Updated by okurz over 3 years ago

  • Copied to action #75274: [osd-admins][alert][learning] Failed systemd services alert (workers): os-autoinst-openvswitch.service aborts retries after 60s and is not easily configurable added
Actions
Copy link
 #7

Updated by okurz over 3 years ago

  • Copied to action #75277: enable "postfix" service again after #73633 resolved added
Actions
Copy link
 #8

Updated by okurz over 3 years ago

called a command to restart all failed systemd services as of now:

sudo salt -l error -C 'G@roles:worker' cmd.run 'sudo systemctl --failed --no-legend | sed "s/\s.*//" | sudo xargs --no-run-if-empty systemctl restart'

And to have a persistent workaround I did:

sudo salt -l error -C 'G@roles:worker' cmd.run 'echo -e "#!/bin/sh\nlogger poo#73633 workarounds\nsleep 300; systemctl --failed --no-legend | sed \"s/\s.*//\" | xargs --no-run-if-empty systemctl restart\nexit 0" > /etc/init.d/boot.local && chmod +x /etc/init.d/boot.local && systemctl enable rc-local'

this should restart all failed services after 5 minutes after startup of machines. After the root cause is fixed we should remove this workaround again.

Actions
Copy link
 #9

Updated by okurz over 3 years ago

So far we have not proceeded much in the according ticket as I could not convince EngInfra personell to investigate the issue themselves further. I gave more specific suggestions in https://infra.nue.suse.com/SelfService/Display.html?id=178694#txn-2643433 , e.g.:

I suspect problems with the switch, potentially also the DHCP server. If possible I would do the following:
* Connect a test notebook to the switch and check if linkup happens within a reasonable time
* Check for correct DHCP response
* Crosscheck if there are any invalid or bogus DHCP packages received, e.g. using tcpdump or wireshark
* Potentially disconnect all other devices connected to the same switch to rule out bogus DHCP server or any hosts conducting an effective, unintended "DoS" attack
* Maybe it is enough to just "reset" or "power cycle" the corresponding switch(es), potentially restart the involved DHCP server
* Disconnect all connections
Actions
Copy link
 #10

Updated by okurz over 3 years ago

wrote in SUSE-IT RC channel: @esujskaja hello, could you help to see that https://infra.nue.suse.com/SelfService/Display.html?id=178694 receives the right priority, please? I marked the ticket with "[openqa][urgent]" and this is actually a big problem for us. I am trying to provide helpful hints as well and be patient but we do not have control over the network infrastructure, your team does. And I would not like to escalate this further (yet).

Actions
Copy link
 #11

Updated by okurz over 3 years ago

Trying to answer mmaher in the ticket:

Unfortunately Nick is currently unavailable.
I assumed all machines are affected, so according to
https://infra.nue.suse.com/SelfService/Display.html?id=178694#txn-2636458
we could pick any machine.

For easier investigation I suggest you add your ssh key to
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/sshd/users.sls
with a merge request. This would allow you easy ssh access to all our OSD worker machines.

Also fghariani and mcaj are already there.

We have two machines which are not used within our production instance "openqaworker11" and "openqaworker12"
but I just checked on openqaworker12 for times between boot and link-up:

openqaworker12:/home/okurz # sudo journalctl --since='2020-09-01' | grep '\(kernel: Linux version\|eth.*Link is Up\|eth.*link becomes ready\)'
Sep 06 03:32:52 localhost kernel: Linux version 4.12.14-lp151.28.63-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Tue Sep 1 11:48:04 UTC 2020 (5009827)
Sep 06 03:33:59 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Sep 06 03:33:59 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 13 03:32:56 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Sep 13 03:34:05 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Sep 13 03:34:05 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 20 03:32:52 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Sep 20 03:34:04 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Sep 20 03:34:04 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 27 03:32:51 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Sep 27 03:34:00 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Sep 27 03:34:00 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 27 03:37:28 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Sep 27 03:38:38 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Sep 27 03:38:38 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 04 03:32:52 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Oct 04 03:34:01 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Oct 04 03:34:01 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 04 03:37:28 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Oct 04 03:38:38 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Oct 04 03:38:38 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 11 03:32:53 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Oct 11 03:34:02 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Oct 11 03:34:02 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 18 03:32:51 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
Oct 18 03:34:01 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Oct 18 03:34:01 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 25 03:32:50 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
Oct 25 03:34:02 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Oct 25 03:34:02 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Nov 01 03:32:52 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
Nov 01 03:34:02 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Nov 01 03:34:02 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Nov 08 03:32:51 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
Nov 08 03:34:02 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Nov 08 03:34:02 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

I see a very consistent 70+-2s over the whole time. So this machine did not change at all. I checked in /srv/salt and the state is 2020-05 so rather old.

Checking on openqaworker11 shows:

openqaworker11:/srv/salt # journalctl --since='2020-09-01' | grep '\(kernel: Linux version\|eth.*Link is Up\|eth.*link becomes ready\)'
Oct 14 19:51:00 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Oct 14 19:52:26 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Oct 14 19:52:26 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 18 03:33:22 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Oct 18 03:34:47 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Oct 18 03:34:47 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Nov 08 03:33:18 localhost kernel: Linux version 4.12.14-lp151.28.67-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Fri Sep 4 15:23:21 UTC 2020 (2c5a14f)
Nov 08 03:34:42 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Nov 08 03:34:42 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

so a consistent 90+-2s. And /srv/salt shows:

commit 57748dedc84d186ebc81d2852586b6e013947f8f (HEAD -> firewall, mkittler/firewall)
Author: Marius Kittler <mkittler@suse.de>
Date:   Tue Oct 13 17:56:20 2020 +0200

    Use firewalld instead of SuSEfirewall2

    See https://progress.opensuse.org/issues/68095

so rather recent and even the change included which we suspected to have caused the issues in #73633

Conducted again on all current OSD machines:

okurz@openqa:~> sudo salt -t 300 \* cmd.run "sudo journalctl --since='2020-09-15' | grep '\(kernel: Linux version\|eth.*Link is Up\|eth.*link becomes ready\)'"
openqaworker2.suse.de:
    Nov 08 03:34:59 linux-1nn1 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:36:17 linux-1nn1 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
openqaworker10.suse.de:
    Nov 08 03:33:48 openqaworker10 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:35:31 openqaworker10 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Nov 08 03:35:31 openqaworker10 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker9.suse.de:
    Nov 08 03:34:00 linux-q6bp kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:35:00 linux-q6bp kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Nov 08 03:35:00 linux-q6bp kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker3.suse.de:
    Nov 08 03:34:36 linux-5h37 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:35:53 linux-5h37 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
openqaworker8.suse.de:
    Nov 08 03:33:58 openqaworker8 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:34:59 openqaworker8 kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
QA-Power8-5-kvm.qa.suse.de:
    Oct 18 03:35:53 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:36:22 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Oct 22 21:43:29 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 22 21:44:14 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Oct 29 14:50:59 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 29 14:52:01 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Oct 29 16:42:10 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 29 16:42:52 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Oct 30 23:04:20 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 30 23:05:03 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Nov 02 17:59:28 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 02 18:00:21 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Nov 08 03:34:45 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:35:29 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
QA-Power8-4-kvm.qa.suse.de:
    Nov 08 03:34:07 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:34:45 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
openqaworker5.suse.de:
    Nov 08 03:36:58 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:38:11 localhost kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Nov 08 03:38:11 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqa.suse.de:
    Nov 10 21:47:45 openqa sudo[18238]:    okurz : TTY=pts/0 ; PWD=/home/okurz ; USER=root ; COMMAND=/usr/bin/salt * cmd.run sudo journalctl --since='2020-10-14' | grep '\(kernel: Linux version\|eth.*Link is Up\|eth.*link becomes ready\)'
    Nov 10 22:11:04 openqa sudo[29318]:    okurz : TTY=pts/0 ; PWD=/home/okurz ; USER=root ; COMMAND=/usr/bin/salt -t 300 * cmd.run sudo journalctl --since='2020-09-15' | grep '\(kernel: Linux version\|eth.*Link is Up\|eth.*link becomes ready\)'
openqaworker6.suse.de:
    Nov 08 03:36:53 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:38:03 localhost kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Nov 08 03:38:03 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
malbec.arch.suse.de:
    Oct 18 03:35:11 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 18 03:36:39 malbec kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
    Oct 22 23:32:44 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 22 23:35:49 malbec kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
    Oct 28 08:36:36 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 28 08:39:39 malbec kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
    Oct 31 09:55:14 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Oct 31 09:58:18 malbec kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
    Nov 03 15:09:18 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 03 15:16:58 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 03 15:20:04 malbec kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
    Nov 08 03:35:24 malbec kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:37:02 malbec kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
openqaworker13.suse.de:
    Nov 08 03:33:41 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 08 03:35:21 localhost kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Nov 08 03:35:21 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
grenache-1.qa.suse.de:
    Nov 08 03:34:42 grenache-1 kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
openqa-monitor.qa.suse.de:
    Nov 08 03:31:39 localhost kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
openqaworker-arm-1.suse.de:
    Oct 24 07:27:59 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 24 07:30:59 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 24 23:44:00 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 24 23:47:01 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 25 10:15:54 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 25 10:18:53 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 25 14:45:10 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 25 14:48:11 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 26 20:05:49 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 26 20:08:48 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 27 18:11:59 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 27 18:15:00 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 28 01:42:50 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 28 01:45:50 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 28 02:47:09 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 28 02:50:09 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 28 03:24:11 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 28 03:29:56 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 28 03:32:57 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 28 09:24:15 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 28 09:27:16 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 28 12:06:07 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 28 12:09:07 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 31 00:11:13 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 31 00:14:11 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 31 06:41:59 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 31 06:44:59 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 31 09:30:02 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 31 09:33:03 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 31 15:30:33 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 31 15:33:34 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Oct 31 16:51:02 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Oct 31 16:54:00 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 02 02:29:07 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 02 02:32:07 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 02 17:36:12 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 02 17:39:11 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 04 01:41:33 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 04 01:44:35 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 06 22:13:12 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 06 22:16:16 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 07 11:57:33 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 07 12:00:41 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 08 03:36:51 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 08 03:42:54 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 08 03:45:55 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 09 17:04:38 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 09 17:07:40 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 10 20:55:56 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 10 20:58:55 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
openqaworker-arm-2.suse.de:
    Minion did not return. [Not connected]
powerqaworker-qam-1:
    Minion did not return. [Not connected]

the first entry, openqaworker2.suse.de, shows a time between kernel boot and link-up of 1m20s which I think is ok (for a server).

malbec.arch.suse.de shows a situation that was ok still on 2020-10-18, worse on 2020-10-22 and later but on 2020-11-08 ok again. openqaworker-arm-1 shows a time of around 3m but showed 1m20s in #75016#note-3 but only from a single data point.

Updated https://infra.nue.suse.com/SelfService/Display.html?id=178694#txn-2665188 accordingly.

Actions
Copy link
 #12

Updated by okurz over 3 years ago

#73633 was closed but https://infra.nue.suse.com/SelfService/Display.html?id=178694#txn-2665188 is still open with suggestions to EngInfra what to do, still keeping "Blocked".

Actions
Copy link
 #13

Updated by livdywan over 3 years ago

  • Related to action #75238: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.2 added
Actions
Copy link
 #14

Updated by okurz over 3 years ago

https://infra.nue.suse.com/SelfService/Display.html?id=178694 was closed without action, reconducted previous check for link-up times:

openqaworker3.suse.de:
    Dec 15 07:27:46 localhost kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 15 07:28:12 openqaworker3 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 15 08:07:08 openqaworker3 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 15 08:12:02 localhost kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 15 08:12:54 openqaworker3 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 20 03:34:31 localhost kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:35:23 openqaworker3 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
openqaworker6.suse.de:
    Dec 13 03:36:57 openqaworker6 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 13 03:38:06 openqaworker6 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 13 03:38:06 openqaworker6 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 20 03:36:56 openqaworker6 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:38:06 openqaworker6 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 20 03:38:06 openqaworker6 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker8.suse.de:
    Dec 09 18:38:33 openqaworker8 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 09 18:39:26 openqaworker8 kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 09 18:39:26 openqaworker8 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 13 03:34:06 openqaworker8 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 13 03:34:59 openqaworker8 kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 13 03:34:59 openqaworker8 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 20 03:34:04 openqaworker8 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:34:57 openqaworker8 kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 20 03:34:57 openqaworker8 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker5.suse.de:
    Dec 13 03:37:02 openqaworker5 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 13 03:38:17 openqaworker5 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 13 03:38:17 openqaworker5 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 20 03:37:01 openqaworker5 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:38:16 openqaworker5 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 20 03:38:16 openqaworker5 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker9.suse.de:
    Dec 13 03:34:08 openqaworker9 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 13 03:35:01 openqaworker9 kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 13 03:35:01 openqaworker9 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 20 03:34:07 openqaworker9 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:35:00 openqaworker9 kernel: igb 0000:03:00.1 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 20 03:35:00 openqaworker9 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
openqaworker2.suse.de:
    Dec 06 03:34:57 openqaworker2 kernel: Linux version 4.12.14-lp151.28.83-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Sat Nov 21 07:05:39 UTC 2020 (85b988f)
    Dec 06 03:36:16 openqaworker2 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 09 19:30:57 openqaworker2 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 09 19:32:10 openqaworker2 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 13 03:34:56 openqaworker2 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 13 03:36:06 openqaworker2 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 14 13:45:42 openqaworker2 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 20 03:34:58 openqaworker2 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:36:09 openqaworker2 kernel: igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
openqaworker13.suse.de:
    Dec 13 03:33:40 openqaworker13 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 13 03:34:55 openqaworker13 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 13 03:34:55 openqaworker13 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 20 03:33:45 openqaworker13 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:35:05 openqaworker13 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 20 03:35:05 openqaworker13 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker10.suse.de:
    Dec 06 03:33:49 openqaworker10 kernel: Linux version 4.12.14-lp151.28.83-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Sat Nov 21 07:05:39 UTC 2020 (85b988f)
    Dec 06 03:35:18 openqaworker10 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 06 03:35:18 openqaworker10 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 09 21:53:00 openqaworker10 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 09 21:54:41 openqaworker10 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 09 21:54:41 openqaworker10 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 13 03:33:50 openqaworker10 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 13 03:35:18 openqaworker10 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 13 03:35:18 openqaworker10 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 20 03:33:51 openqaworker10 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:35:21 openqaworker10 kernel: igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    Dec 20 03:35:21 openqaworker10 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqa.suse.de:
    Dec 20 03:30:41 openqa kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 21 07:09:14 openqa sudo[9138]:    okurz : TTY=pts/0 ; PWD=/home/okurz ; USER=root ; COMMAND=/usr/bin/salt -t 300 * cmd.run sudo journalctl --since='2020-09-15' | grep '\(kernel: Linux version\|eth.*Link is Up\|eth.*link becomes ready\)'
QA-Power8-4-kvm.qa.suse.de:
    Dec 06 03:34:06 QA-Power8-4-kvm kernel: Linux version 4.12.14-lp151.28.83-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Sat Nov 21 07:05:39 UTC 2020 (85b988f)
    Dec 09 23:25:44 QA-Power8-4-kvm kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 10 10:22:40 QA-Power8-4-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 14 08:51:14 QA-Power8-4-kvm kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 14 08:52:00 QA-Power8-4-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 14 09:30:08 QA-Power8-4-kvm kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 14 09:30:47 QA-Power8-4-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 14 13:04:50 QA-Power8-4-kvm kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 14 13:05:33 QA-Power8-4-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 14 13:33:04 QA-Power8-4-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
QA-Power8-5-kvm.qa.suse.de:
    Nov 11 09:09:11 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 11 09:10:11 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Nov 16 13:03:04 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.75-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Thu Oct 15 12:09:20 UTC 2020 (3577f5a)
    Nov 16 13:03:55 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Nov 19 06:29:06 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.79-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Wed Nov 11 08:17:16 UTC 2020 (472d149)
    Nov 19 06:29:49 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Nov 24 12:47:33 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.79-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Wed Nov 11 08:17:16 UTC 2020 (472d149)
    Nov 24 12:48:19 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Nov 27 09:24:22 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.79-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Wed Nov 11 08:17:16 UTC 2020 (472d149)
    Nov 27 09:25:06 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 01 11:54:47 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.83-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Sat Nov 21 07:05:39 UTC 2020 (85b988f)
    Dec 01 11:56:05 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 03 10:03:30 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.83-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Sat Nov 21 07:05:39 UTC 2020 (85b988f)
    Dec 03 10:04:10 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 08 09:18:12 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.83-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Sat Nov 21 07:05:39 UTC 2020 (85b988f)
    Dec 08 09:18:59 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 09 11:27:46 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.87-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Wed Dec 2 07:20:50 UTC 2020 (bd14900)
    Dec 09 11:28:30 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 09 16:01:22 QA-Power8-5-kvm kernel: Linux version 4.12.14-lp151.28.87-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux) ) #1 SMP Wed Dec 2 07:20:50 UTC 2020 (bd14900)
    Dec 09 16:02:05 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 10 10:38:32 QA-Power8-5-kvm kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 10 10:39:40 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
    Dec 14 13:15:26 QA-Power8-5-kvm kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 14 13:16:40 QA-Power8-5-kvm kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
grenache-1.qa.suse.de:
    Dec 09 22:40:18 grenache-1 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 13 03:34:43 grenache-1 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
    Dec 20 03:34:40 grenache-1 kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
openqa-monitor.qa.suse.de:
    Dec 20 03:30:17 openqa-monitor kernel: Linux version 5.3.18-lp152.57-default (geeko@buildhost) (gcc version 7.5.0 (SUSE Linux)) #1 SMP Fri Dec 4 07:27:58 UTC 2020 (7be5551)
openqaworker-arm-1.suse.de:
    Nov 26 11:24:08 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 26 11:29:54 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 26 11:32:57 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 26 12:52:59 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 26 12:55:59 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 26 13:29:58 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 26 13:32:56 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 26 14:59:47 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 26 15:05:44 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 26 15:08:47 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Nov 28 01:05:57 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Nov 28 01:08:57 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 02 18:50:57 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 02 18:53:57 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 04 12:07:20 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 04 12:10:20 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 04 17:36:02 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 04 17:39:03 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 05 00:31:54 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 05 00:34:54 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 08 23:21:01 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 08 23:24:03 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 11 19:03:00 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 11 19:06:05 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 11 19:06:05 openqaworker-arm-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 16 13:01:58 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 16 13:05:02 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 16 13:05:02 openqaworker-arm-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 18 03:10:52 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 18 03:13:51 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 18 03:13:51 openqaworker-arm-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 18 16:42:51 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 18 16:45:49 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 18 16:45:49 openqaworker-arm-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    Dec 20 03:36:15 openqaworker-arm-1 kernel: Linux version 5.8.3-1.gbad027a-default (geeko@buildhost) (gcc (SUSE Linux) 10.2.1 20200805 [revision dda1e9d08434def88ed86557d08b23251332c5aa], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 22 06:31:07 UTC 2020 (bad027a)
    Dec 20 03:39:14 openqaworker-arm-1 kernel: nicvf 0002:01:00.1 eth0: Link is Up 10000 Mbps Full duplex
    Dec 20 03:39:14 openqaworker-arm-1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
openqaworker-arm-2.suse.de:
    Dec 07 03:59:07 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 07 04:03:09 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 12 09:46:04 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 12 09:50:04 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 12 09:50:04 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 14 09:08:33 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 14 09:12:04 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 14 09:12:04 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 14 12:53:59 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 14 12:53:59 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 14 21:46:00 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 14 21:49:39 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 14 21:49:39 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 15 00:53:43 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 15 00:57:19 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 15 00:57:19 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 15 01:29:01 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 15 01:32:36 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 15 01:32:36 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 15 04:41:23 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 15 04:44:59 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 15 04:44:59 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 16 16:05:03 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 16 16:08:39 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 16 16:08:39 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 17 10:54:20 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 17 10:57:55 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 17 10:57:55 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 17 11:16:45 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 17 11:20:37 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 17 11:24:10 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 17 11:24:10 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 17 19:59:00 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 17 20:02:34 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 17 20:02:34 openqaworker-arm-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    Dec 18 14:44:21 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 18 14:47:50 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
    Dec 19 17:16:00 openqaworker-arm-2 kernel: Linux version 5.7.12-1.g9c98feb-default (geeko@buildhost) (gcc version 10.1.1 20200625 [revision c91e43e9363bd119a695d64505f96539fa451bf2] (SUSE Linux), GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.34.0.20200325-1) #1 SMP Sat Aug 1 06:18:58 UTC 2020 (9c98feb)
    Dec 19 17:19:28 openqaworker-arm-2 kernel: nicvf 0002:01:00.1 eth1: Link is Up 10000 Mbps Full duplex
malbec.arch.suse.de:
    Minion did not return. [Not connected]
ERROR: Minions returned with non-zero exit code

also checked on openqaworker-arm-3 which is not currently in salt. It seems all machines have a link-up within 1-2m except for arm machines that take 3-5m which could be an arm specific firmware problem or something else. Likely not high chance that we would solve this over "EngInfra".

I have provided an update in the EngInfra ticket with the explanation that likely the forced power cycle of the network switches in 2020-11 SUSE nbg data center might have mitigated the issue enough.

Checking for currently failing systemd services within OSD I found a lot.

Actions
Copy link
 #15

Updated by okurz about 3 years ago

  • Status changed from Blocked to Workable
  • Assignee deleted (okurz)

AFter some time has passed we should check again for any failed services and if none left then re-enable the alerts.

Actions
Copy link
 #16

Updated by mkittler about 3 years ago

There were still lots of failed systemd services: https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&editPanel=6&tab=alert&from=now-90d&to=now

One can not tell which services were responsible and hence one can not tell whether they are related to that ticket. Note that there are currently two unrelated failed services:

powerqaworker-qam-1:
    logrotate.service loaded failed failed Rotate log files
grenache-1.qa.suse.de:
    logrotate.service loaded failed failed Rotate log files
Actions
Copy link
 #17

Updated by okurz about 3 years ago

  • Subject changed from [osd-admins][alert] Failed systemd services alert (workers): os-autoinst-openvswitch.service (and var-lib-openqa-share.mount) on openqaworker-arm-2 to [osd-admins][alert] Failed systemd services alert (workers): os-autoinst-openvswitch.service (and var-lib-openqa-share.mount) on openqaworker-arm-2 and others
Actions
Copy link
 #18

Updated by okurz about 3 years ago

  • Priority changed from High to Normal

I suggest we work on #89551 first which could be related to some problems on bootup

Actions
Copy link
 #19

Updated by okurz about 3 years ago

  • Related to action #89551: NFS mount fails after boot (reproducible on some OSD workers) added
Actions
Copy link
 #20

Updated by mkittler about 3 years ago

  • Priority changed from Normal to High

I created #89551 for the NFS-specific problem (which occurs independently of os-autoinst-openvswitch.service).

Actions
Copy link
 #21

Updated by okurz about 3 years ago

  • Status changed from Workable to Blocked
  • Assignee set to mkittler

let's work on #89551 first so please report back here after that is done.

Actions
Copy link
 #22

Updated by okurz about 3 years ago

  • Status changed from Blocked to Workable

unblocked

Actions
Copy link
 #23

Updated by mkittler about 3 years ago

  • Status changed from Workable to Resolved

I haven't seen failing os-autoinst-openvswitch.service in a while and for var-lib-openqa-share.mount see https://progress.opensuse.org/issues/89551#change-394019.

Actions
Copy link
 #24

Updated by okurz about 1 month ago

Actions
Copy link

Also available in: Atom PDF