action #97532
[qe-core][sporadic] s390x jobs are failing to boot auto_review:"error: Cannot set interface flags on 'macvtap.*': Address already in use":retry
0%
Description
Observation¶
390x jobs on grenache-1:44 are still failing to boot: https://openqa.suse.de/tests/6926261#step/bootloader_zkvm/28
error: Failed to start domain openQA-SUT-5 error: Cannot set interface flags on 'macvtap2': Address already in use
this might be if there is some manually maintained instance left over blocking the ressources.
Suggestions¶
- Check ressources on s390x KVM host
- Find out how to map network devices to VMs here
Related issues
History
#1
Updated by tjyrinki_suse almost 2 years ago
- Status changed from New to Workable
- Priority changed from High to Urgent
- Target version set to QE-Core: Ready
- Start date deleted (
2021-08-26)
#2
Updated by tjyrinki_suse almost 2 years ago
At least on the example's host (s390zp19) there are no stray/old instances running, nor are there left around macvtap interfaces.
#3
Updated by tjyrinki_suse almost 2 years ago
- Priority changed from Urgent to High
Some s390x people are on holidays so the network device mapping investigation may need to wait, but at least there does not seem to be anything particularly wrong on the kvm host itself. Sporadic failing is annoying though since it could calse failures for any s390x tests.
#4
Updated by szarate almost 2 years ago
- Related to action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process added
#5
Updated by szarate almost 2 years ago
I have left this scrip running on that machine to collect some data at the moment this is happening, use screen -x 75736.virshmonitor_screen
to attach to it
should give a rough idea of what's going on at the time this happens (I'm guessing leftover qemu process)
Next step if happens again, enable debug
#6
Updated by szarate over 1 year ago
Well, we're here and happened again for some other reason:
Sep 06 11:31:07 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap4': Address already in use 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::1488:79ff:fe53:d9ce/64 scope link valid_lft forever preferred_lft forever 51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fea6:258a/64 scope link valid_lft forever preferred_lft forever 54: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe12:5cd6/64 scope link valid_lft forever preferred_lft forever 55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febb:c143/64 scope link valid_lft forever preferred_lft forever 56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe8f:b923/64 scope link valid_lft forever preferred_lft forever 58: macvtap4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500 link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12 openQA Instance 1: 07032013-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_cve@s390x-kvm-sle12 openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12 openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12 openQA Instance 5: 07033363-sle-15-SP2-Server-DVD-HA-Incidents-Install-s390x-Build:21035:haproxy-qam-incidentinstall-ha@zkvm openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12 Sep 06 11:38:26 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap2': Address already in use 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::1488:79ff:fe53:d9ce/64 scope link valid_lft forever preferred_lft forever 51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fea6:258a/64 scope link valid_lft forever preferred_lft forever 55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febb:c143/64 scope link valid_lft forever preferred_lft forever 56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe8f:b923/64 scope link valid_lft forever preferred_lft forever 59: macvtap2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500 link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12 openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12 openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12 openQA Instance 5: 07033373-sle-12-SP4-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-111.1.g7935874-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12 openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12 Sep 06 11:43:15 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap7': Address already in use 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::1488:79ff:fe53:d9ce/64 scope link valid_lft forever preferred_lft forever 51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fea6:258a/64 scope link valid_lft forever preferred_lft forever 55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febb:c143/64 scope link valid_lft forever preferred_lft forever 56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe8f:b923/64 scope link valid_lft forever preferred_lft forever 60: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:7b:c2:33 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe7b:c233/64 scope link valid_lft forever preferred_lft forever 61: macvtap4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:55:e3:d2 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe55:e3d2/64 scope link valid_lft forever preferred_lft forever 62: macvtap5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:45:3a:6c brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe45:3a6c/64 scope link valid_lft forever preferred_lft forever 63: macvtap6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe12:5cd6/64 scope link valid_lft forever preferred_lft forever 64: macvtap7@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500 link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12 openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12 openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12 openQA Instance 10: 07033402-sle-15-SP2-Server-DVD-Incidents-Kernel-KOTD-s390x-Build5.3.18-327.1.g9a3c219-ltp_cve_git@s390x-kvm-sle12 openQA Instance 6: 07033393-sle-15-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-148.1.g89b3900-ltp_cve_git@s390x-kvm-sle12 openQA Instance 7: 07033386-sle-15-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-148.1.g89b3900-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12 openQA Instance 1: 07033403-sle-15-SP3-Server-DVD-Incidents-Kernel-KOTD-s390x-Build5.3.18-289.1.gfb75ef9-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12 openQA Instance 5: 07033413-sle-12-SP4-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-111.1.g7935874-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12 openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12
#7
Updated by szarate over 1 year ago
Ok, so the problem is that some mac addresses are being recycled somehow:
52:54:00:8f:b9:23
in this case
BEGIN PROCESSING: Sep 10 10:37:00 s390zp19 libvirtd[1700]: Cannot set interface flags on 'macvtap7': Address already in use 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::1488:79ff:fe53:d9ce/64 scope link valid_lft forever preferred_lft forever 294: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:62:4d:16 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe62:4d16/64 scope link valid_lft forever preferred_lft forever 296: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe8f:b923/64 scope link valid_lft forever preferred_lft forever 298: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe12:5cd6/64 scope link valid_lft forever preferred_lft forever 299: macvtap4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:7b:c2:33 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe7b:c233/64 scope link valid_lft forever preferred_lft forever 300: macvtap6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:55:e3:d2 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe55:e3d2/64 scope link valid_lft forever preferred_lft forever 301: macvtap5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:7f:c6:11 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe7f:c611/64 scope link valid_lft forever preferred_lft forever 304: macvtap9@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:45:3a:6c brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:fe45:3a6c/64 scope link valid_lft forever preferred_lft forever 305: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febb:c143/64 scope link valid_lft forever preferred_lft forever openQA Instance 3: 07071743-sle-12-SP5-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests2@s390x-kvm-sle15 Interface Type Source Model MAC ----------------------------------------------------------- macvtap2 direct eth0 virtio 52:54:00:62:4d:16 1 openQA Instance 64: 00006848-sle-micro-5.1-DVD-s390x-Build60.1_12.19-sle-micro_installation+update@s390x-kvm-sle12 Interface Type Source Model MAC ----------------------------------------------------------- macvtap3 direct eth0 virtio 52:54:00:8f:b9:23 1 openQA Instance 1: 07071585-sle-12-SP4-Server-DVD-Updates-s390x-Build20210910-2-ltp_syscalls@s390x-kvm-sle12 Interface Type Source Model MAC ----------------------------------------------------------- macvtap0 direct eth0 virtio 52:54:00:12:5c:d6 1 openQA Instance 10: 07071587-sle-12-SP4-Server-DVD-Updates-s390x-Build20210910-2-ltp_cve@s390x-kvm-sle12 Interface Type Source Model MAC ----------------------------------------------------------- macvtap4 direct eth0 virtio 52:54:00:7b:c2:33 1 openQA Instance 6: 07073191-sle-12-SP5-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests1@s390x-kvm-sle15 Interface Type Source Model MAC ----------------------------------------------------------- macvtap6 direct eth0 virtio 52:54:00:55:e3:d2 1 openQA Instance 4: 07071966-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests2@s390x-kvm-sle12 Interface Type Source Model MAC ----------------------------------------------------------- macvtap5 direct eth0 virtio 52:54:00:7f:c6:11 1 openQA Instance 7: 07071970-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-mau-bind@s390x-kvm-sle12 Interface Type Source Model MAC ----------------------------------------------------------- macvtap9 direct eth0 virtio 52:54:00:45:3a:6c 1 openQA Instance 9: 07071976-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-sssd_openldap_functional@s390x-kvm-sle12 Interface Type Source Model MAC ----------------------------------------------------------- macvtap1 direct eth0 virtio 52:54:00:bb:c1:43 1 openQA Instance 5: 07071977-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-sssd_389ds_functional@s390x-kvm-sle12 Interface Type Source Model MAC ----------------------------------------------------------- - direct eth0 virtio 52:54:00:8f:b9:23 1 FINISH PROCESSING: Sep 10 10:37:00 s390zp19 libvirtd[1700]: Cannot set interface flags on 'macvtap7': Address already in use
#8
Updated by szarate over 1 year ago
- Related to action #98541: [qe-core][kernel] Steps in case of s390 failures added
#9
Updated by okurz over 1 year ago
Please consider picking up this ticket within the next 30 days or just set the ticket to the next lower priority of "Normal" (SLO: updated within 365 days).
#10
Updated by szarate over 1 year ago
- Status changed from Workable to Resolved
- Assignee set to szarate
I will call this done, as there's no more errors reported in that case...
I do see
error: Failed to start domain openQA-SUT-8 error: cannot connect to netlink socket with protocol 0: Address already in use
On this job: https://openqa.suse.de/tests/7332379# but since it happened only once, I'll just ask in discuss-virtualization how that can be improved and Will see later what to do.
#11
Updated by szarate over 1 year ago
- Precedes action #100841: Improve logging for libvirt hosts added
#12
Updated by szarate over 1 year ago
- Precedes deleted (action #100841: Improve logging for libvirt hosts)
#13
Updated by szarate over 1 year ago
- Precedes action #100841: Improve logging for libvirt hosts added