Actions
action #97532
closed[qe-core][sporadic] s390x jobs are failing to boot auto_review:"error: Cannot set interface flags on 'macvtap.*': Address already in use":retry
Added by okurz about 3 years ago. Updated about 3 years ago.
Status:
Resolved
Priority:
High
Assignee:
Category:
Infrastructure
Target version:
Start date:
Due date:
% Done:
0%
Estimated time:
Difficulty:
Description
Observation¶
390x jobs on grenache-1:44 are still failing to boot: https://openqa.suse.de/tests/6926261#step/bootloader_zkvm/28
error: Failed to start domain openQA-SUT-5
error: Cannot set interface flags on 'macvtap2': Address already in use
this might be if there is some manually maintained instance left over blocking the ressources.
Suggestions¶
- Check ressources on s390x KVM host
- Find out how to map network devices to VMs here
Updated by tjyrinki_suse about 3 years ago
- Status changed from New to Workable
- Priority changed from High to Urgent
- Target version set to QE-Core: Ready
- Start date deleted (
2021-08-26)
Updated by tjyrinki_suse about 3 years ago
At least on the example's host (s390zp19) there are no stray/old instances running, nor are there left around macvtap interfaces.
Updated by tjyrinki_suse about 3 years ago
- Priority changed from Urgent to High
Some s390x people are on holidays so the network device mapping investigation may need to wait, but at least there does not seem to be anything particularly wrong on the kvm host itself. Sporadic failing is annoying though since it could calse failures for any s390x tests.
Updated by szarate about 3 years ago
- Related to action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process added
Updated by szarate about 3 years ago
I have left this scrip running on that machine to collect some data at the moment this is happening, use screen -x 75736.virshmonitor_screen to attach to it
should give a rough idea of what's going on at the time this happens (I'm guessing leftover qemu process)
Next step if happens again, enable debug
Updated by szarate about 3 years ago
Well, we're here and happened again for some other reason:
Sep 06 11:31:07 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap4': Address already in use
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fea6:258a/64 scope link
valid_lft forever preferred_lft forever
54: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe12:5cd6/64 scope link
valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:febb:c143/64 scope link
valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe8f:b923/64 scope link
valid_lft forever preferred_lft forever
58: macvtap4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12
openQA Instance 1: 07032013-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_cve@s390x-kvm-sle12
openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12
openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12
openQA Instance 5: 07033363-sle-15-SP2-Server-DVD-HA-Incidents-Install-s390x-Build:21035:haproxy-qam-incidentinstall-ha@zkvm
openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12
Sep 06 11:38:26 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap2': Address already in use
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fea6:258a/64 scope link
valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:febb:c143/64 scope link
valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe8f:b923/64 scope link
valid_lft forever preferred_lft forever
59: macvtap2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12
openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12
openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12
openQA Instance 5: 07033373-sle-12-SP4-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-111.1.g7935874-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12
openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12
Sep 06 11:43:15 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap7': Address already in use
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fea6:258a/64 scope link
valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:febb:c143/64 scope link
valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe8f:b923/64 scope link
valid_lft forever preferred_lft forever
60: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:7b:c2:33 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe7b:c233/64 scope link
valid_lft forever preferred_lft forever
61: macvtap4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:55:e3:d2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe55:e3d2/64 scope link
valid_lft forever preferred_lft forever
62: macvtap5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:45:3a:6c brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe45:3a6c/64 scope link
valid_lft forever preferred_lft forever
63: macvtap6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe12:5cd6/64 scope link
valid_lft forever preferred_lft forever
64: macvtap7@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12
openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12
openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12
openQA Instance 10: 07033402-sle-15-SP2-Server-DVD-Incidents-Kernel-KOTD-s390x-Build5.3.18-327.1.g9a3c219-ltp_cve_git@s390x-kvm-sle12
openQA Instance 6: 07033393-sle-15-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-148.1.g89b3900-ltp_cve_git@s390x-kvm-sle12
openQA Instance 7: 07033386-sle-15-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-148.1.g89b3900-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12
openQA Instance 1: 07033403-sle-15-SP3-Server-DVD-Incidents-Kernel-KOTD-s390x-Build5.3.18-289.1.gfb75ef9-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12
openQA Instance 5: 07033413-sle-12-SP4-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-111.1.g7935874-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12
openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12
Updated by szarate about 3 years ago
Ok, so the problem is that some mac addresses are being recycled somehow:
52:54:00:8f:b9:23 in this case
BEGIN PROCESSING: Sep 10 10:37:00 s390zp19 libvirtd[1700]: Cannot set interface flags on 'macvtap7': Address already in use
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
valid_lft forever preferred_lft forever
294: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:62:4d:16 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe62:4d16/64 scope link
valid_lft forever preferred_lft forever
296: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe8f:b923/64 scope link
valid_lft forever preferred_lft forever
298: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe12:5cd6/64 scope link
valid_lft forever preferred_lft forever
299: macvtap4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:7b:c2:33 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe7b:c233/64 scope link
valid_lft forever preferred_lft forever
300: macvtap6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:55:e3:d2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe55:e3d2/64 scope link
valid_lft forever preferred_lft forever
301: macvtap5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:7f:c6:11 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe7f:c611/64 scope link
valid_lft forever preferred_lft forever
304: macvtap9@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:45:3a:6c brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe45:3a6c/64 scope link
valid_lft forever preferred_lft forever
305: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:febb:c143/64 scope link
valid_lft forever preferred_lft forever
openQA Instance 3: 07071743-sle-12-SP5-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests2@s390x-kvm-sle15
Interface Type Source Model MAC
-----------------------------------------------------------
macvtap2 direct eth0 virtio 52:54:00:62:4d:16
1
openQA Instance 64: 00006848-sle-micro-5.1-DVD-s390x-Build60.1_12.19-sle-micro_installation+update@s390x-kvm-sle12
Interface Type Source Model MAC
-----------------------------------------------------------
macvtap3 direct eth0 virtio 52:54:00:8f:b9:23
1
openQA Instance 1: 07071585-sle-12-SP4-Server-DVD-Updates-s390x-Build20210910-2-ltp_syscalls@s390x-kvm-sle12
Interface Type Source Model MAC
-----------------------------------------------------------
macvtap0 direct eth0 virtio 52:54:00:12:5c:d6
1
openQA Instance 10: 07071587-sle-12-SP4-Server-DVD-Updates-s390x-Build20210910-2-ltp_cve@s390x-kvm-sle12
Interface Type Source Model MAC
-----------------------------------------------------------
macvtap4 direct eth0 virtio 52:54:00:7b:c2:33
1
openQA Instance 6: 07073191-sle-12-SP5-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests1@s390x-kvm-sle15
Interface Type Source Model MAC
-----------------------------------------------------------
macvtap6 direct eth0 virtio 52:54:00:55:e3:d2
1
openQA Instance 4: 07071966-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests2@s390x-kvm-sle12
Interface Type Source Model MAC
-----------------------------------------------------------
macvtap5 direct eth0 virtio 52:54:00:7f:c6:11
1
openQA Instance 7: 07071970-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-mau-bind@s390x-kvm-sle12
Interface Type Source Model MAC
-----------------------------------------------------------
macvtap9 direct eth0 virtio 52:54:00:45:3a:6c
1
openQA Instance 9: 07071976-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-sssd_openldap_functional@s390x-kvm-sle12
Interface Type Source Model MAC
-----------------------------------------------------------
macvtap1 direct eth0 virtio 52:54:00:bb:c1:43
1
openQA Instance 5: 07071977-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-sssd_389ds_functional@s390x-kvm-sle12
Interface Type Source Model MAC
-----------------------------------------------------------
- direct eth0 virtio 52:54:00:8f:b9:23
1
FINISH PROCESSING: Sep 10 10:37:00 s390zp19 libvirtd[1700]: Cannot set interface flags on 'macvtap7': Address already in use
Updated by szarate about 3 years ago
- Related to action #98541: [qe-core][kernel] Steps in case of s390 failures added
Updated by okurz about 3 years ago
Please consider picking up this ticket within the next 30 days or just set the ticket to the next lower priority of "Normal" (SLO: updated within 365 days).
Updated by szarate about 3 years ago
- Status changed from Workable to Resolved
- Assignee set to szarate
I will call this done, as there's no more errors reported in that case...
I do see
error: Failed to start domain openQA-SUT-8
error: cannot connect to netlink socket with protocol 0: Address already in use
On this job: https://openqa.suse.de/tests/7332379# but since it happened only once, I'll just ask in discuss-virtualization how that can be improved and Will see later what to do.
Updated by szarate about 3 years ago
- Precedes action #100841: Improve logging for libvirt hosts added
Updated by szarate about 3 years ago
- Precedes deleted (action #100841: Improve logging for libvirt hosts)
Updated by szarate about 3 years ago
- Precedes action #100841: Improve logging for libvirt hosts added
Actions