action #97532: [qe-core][sporadic] s390x jobs are failing to boot auto_review:"error: Cannot set interface flags on 'macvtap.*': Address already in use":retry - openQA Tests - openSUSE Project Management Tool

action #97532

[qe-core][sporadic] s390x jobs are failing to boot auto_review:"error: Cannot set interface flags on 'macvtap.*': Address already in use":retry

Added by okurz almost 2 years ago. Updated over 1 year ago.

Status:

Resolved

Priority:

High

Assignee:

szarate

Category:

Infrastructure

Target version:

QA - QE-Core: Ready

Start date:

Due date:

% Done:

Estimated time:

Difficulty:

Description

Observation¶

390x jobs on grenache-1:44 are still failing to boot: https://openqa.suse.de/tests/6926261#step/bootloader_zkvm/28

error: Failed to start domain openQA-SUT-5
error: Cannot set interface flags on 'macvtap2': Address already in use

this might be if there is some manually maintained instance left over blocking the ressources.

Suggestions¶

Check ressources on s390x KVM host
Find out how to map network devices to VMs here

Related issues

History

#1 Updated by tjyrinki_suse almost 2 years ago

Status changed from New to Workable
Priority changed from High to Urgent
Target version set to QE-Core: Ready
Start date deleted (~~2021-08-26~~)

#2 Updated by tjyrinki_suse almost 2 years ago

At least on the example's host (s390zp19) there are no stray/old instances running, nor are there left around macvtap interfaces.

#3 Updated by tjyrinki_suse almost 2 years ago

Priority changed from Urgent to High

Some s390x people are on holidays so the network device mapping investigation may need to wait, but at least there does not seem to be anything particularly wrong on the kvm host itself. Sporadic failing is annoying though since it could calse failures for any s390x tests.

#4 Updated by szarate almost 2 years ago

Related to action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process added

#5 Updated by szarate almost 2 years ago

I have left this scrip running on that machine to collect some data at the moment this is happening, use screen -x 75736.virshmonitor_screen to attach to it

should give a rough idea of what's going on at the time this happens (I'm guessing leftover qemu process)

Next step if happens again, enable debug

#6 Updated by szarate over 1 year ago

Well, we're here and happened again for some other reason:

Sep 06 11:31:07 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap4': Address already in use

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
    inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
       valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fea6:258a/64 scope link
       valid_lft forever preferred_lft forever
54: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe12:5cd6/64 scope link
       valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febb:c143/64 scope link
       valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe8f:b923/64 scope link
       valid_lft forever preferred_lft forever
58: macvtap4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff

openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 1: 07032013-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_cve@s390x-kvm-sle12

openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12

openQA Instance 5: 07033363-sle-15-SP2-Server-DVD-HA-Incidents-Install-s390x-Build:21035:haproxy-qam-incidentinstall-ha@zkvm

openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12

Sep 06 11:38:26 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap2': Address already in use

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
    inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
       valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fea6:258a/64 scope link
       valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febb:c143/64 scope link
       valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe8f:b923/64 scope link
       valid_lft forever preferred_lft forever
59: macvtap2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff

openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12

openQA Instance 5: 07033373-sle-12-SP4-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-111.1.g7935874-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12

openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12

Sep 06 11:43:15 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap7': Address already in use

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
    inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
       valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fea6:258a/64 scope link
       valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febb:c143/64 scope link
       valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe8f:b923/64 scope link
       valid_lft forever preferred_lft forever
60: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:7b:c2:33 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7b:c233/64 scope link
       valid_lft forever preferred_lft forever
61: macvtap4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:55:e3:d2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe55:e3d2/64 scope link
       valid_lft forever preferred_lft forever
62: macvtap5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:45:3a:6c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe45:3a6c/64 scope link
       valid_lft forever preferred_lft forever
63: macvtap6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe12:5cd6/64 scope link
       valid_lft forever preferred_lft forever
64: macvtap7@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff

openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12

openQA Instance 10: 07033402-sle-15-SP2-Server-DVD-Incidents-Kernel-KOTD-s390x-Build5.3.18-327.1.g9a3c219-ltp_cve_git@s390x-kvm-sle12

openQA Instance 6: 07033393-sle-15-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-148.1.g89b3900-ltp_cve_git@s390x-kvm-sle12

openQA Instance 7: 07033386-sle-15-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-148.1.g89b3900-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12

openQA Instance 1: 07033403-sle-15-SP3-Server-DVD-Incidents-Kernel-KOTD-s390x-Build5.3.18-289.1.gfb75ef9-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12

openQA Instance 5: 07033413-sle-12-SP4-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-111.1.g7935874-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12

openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12

#7 Updated by szarate over 1 year ago

Ok, so the problem is that some mac addresses are being recycled somehow:

52:54:00:8f:b9:23 in this case

BEGIN PROCESSING:   Sep 10 10:37:00 s390zp19 libvirtd[1700]: Cannot set interface flags on 'macvtap7': Address already in use

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
    inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1488:79ff:fe53:d9ce/64 scope link 
       valid_lft forever preferred_lft forever
294: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:62:4d:16 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe62:4d16/64 scope link 
       valid_lft forever preferred_lft forever
296: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe8f:b923/64 scope link 
       valid_lft forever preferred_lft forever
298: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe12:5cd6/64 scope link 
       valid_lft forever preferred_lft forever
299: macvtap4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:7b:c2:33 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7b:c233/64 scope link 
       valid_lft forever preferred_lft forever
300: macvtap6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:55:e3:d2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe55:e3d2/64 scope link 
       valid_lft forever preferred_lft forever
301: macvtap5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:7f:c6:11 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7f:c611/64 scope link 
       valid_lft forever preferred_lft forever
304: macvtap9@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:45:3a:6c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe45:3a6c/64 scope link 
       valid_lft forever preferred_lft forever
305: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febb:c143/64 scope link 
       valid_lft forever preferred_lft forever

openQA Instance 3: 07071743-sle-12-SP5-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests2@s390x-kvm-sle15

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap2    direct   eth0     virtio   52:54:00:62:4d:16


1
openQA Instance 64: 00006848-sle-micro-5.1-DVD-s390x-Build60.1_12.19-sle-micro_installation+update@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap3    direct   eth0     virtio   52:54:00:8f:b9:23


1
openQA Instance 1: 07071585-sle-12-SP4-Server-DVD-Updates-s390x-Build20210910-2-ltp_syscalls@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap0    direct   eth0     virtio   52:54:00:12:5c:d6


1
openQA Instance 10: 07071587-sle-12-SP4-Server-DVD-Updates-s390x-Build20210910-2-ltp_cve@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap4    direct   eth0     virtio   52:54:00:7b:c2:33


1
openQA Instance 6: 07073191-sle-12-SP5-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests1@s390x-kvm-sle15

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap6    direct   eth0     virtio   52:54:00:55:e3:d2


1
openQA Instance 4: 07071966-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests2@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap5    direct   eth0     virtio   52:54:00:7f:c6:11


1
openQA Instance 7: 07071970-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-mau-bind@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap9    direct   eth0     virtio   52:54:00:45:3a:6c


1
openQA Instance 9: 07071976-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-sssd_openldap_functional@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap1    direct   eth0     virtio   52:54:00:bb:c1:43


1
openQA Instance 5: 07071977-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-sssd_389ds_functional@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 -           direct   eth0     virtio   52:54:00:8f:b9:23


1
FINISH PROCESSING:  Sep 10 10:37:00 s390zp19 libvirtd[1700]: Cannot set interface flags on 'macvtap7': Address already in use

#8 Updated by szarate over 1 year ago

Related to action #98541: [qe-core][kernel] Steps in case of s390 failures added

#9 Updated by okurz over 1 year ago

Please consider picking up this ticket within the next 30 days or just set the ticket to the next lower priority of "Normal" (SLO: updated within 365 days).

#10 Updated by szarate over 1 year ago

Status changed from Workable to Resolved
Assignee set to szarate

I will call this done, as there's no more errors reported in that case...

I do see

error: Failed to start domain openQA-SUT-8
error: cannot connect to netlink socket with protocol 0: Address already in use

On this job: https://openqa.suse.de/tests/7332379# but since it happened only once, I'll just ask in discuss-virtualization how that can be improved and Will see later what to do.

#11 Updated by szarate over 1 year ago

Precedes action #100841: Improve logging for libvirt hosts added

#12 Updated by szarate over 1 year ago

Precedes deleted (action #100841: Improve logging for libvirt hosts)

#13 Updated by szarate over 1 year ago

Precedes action #100841: Improve logging for libvirt hosts added

Also available in: Atom PDF

Project

General

Profile

QA » openQA Project » openQA Tests

Issues

Tags

Custom queries