Project

General

Profile

Actions
Copy link

action #97532

closed

[qe-core][sporadic] s390x jobs are failing to boot auto_review:"error: Cannot set interface flags on 'macvtap.*': Address already in use":retry

Added by okurz about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
szarate
Category:
Infrastructure
Target version:
QA - QE-Core: Ready
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

390x jobs on grenache-1:44 are still failing to boot: https://openqa.suse.de/tests/6926261#step/bootloader_zkvm/28

error: Failed to start domain openQA-SUT-5
error: Cannot set interface flags on 'macvtap2': Address already in use

this might be if there is some manually maintained instance left over blocking the ressources.

Suggestions

  • Check ressources on s390x KVM host
  • Find out how to map network devices to VMs here

Related issues 3 (1 open — 2 closed)

Related to openQA Tests - action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu processResolvedszarate2021-01-26

Actions
Related to openQA Tests - action #98541: [qe-core][kernel] Steps in case of s390 failuresResolved2021-09-13

Actions
Precedes openQA Infrastructure - action #100841: Improve logging for libvirt hostsNewszarate2021-10-12

Actions
Actions
Copy link
 #1

Updated by tjyrinki_suse about 3 years ago

  • Status changed from New to Workable
  • Priority changed from High to Urgent
  • Target version set to QE-Core: Ready
  • Start date deleted (2021-08-26)
Actions
Copy link
 #2

Updated by tjyrinki_suse about 3 years ago

At least on the example's host (s390zp19) there are no stray/old instances running, nor are there left around macvtap interfaces.

Actions
Copy link
 #3

Updated by tjyrinki_suse about 3 years ago

  • Priority changed from Urgent to High

Some s390x people are on holidays so the network device mapping investigation may need to wait, but at least there does not seem to be anything particularly wrong on the kvm host itself. Sporadic failing is annoying though since it could calse failures for any s390x tests.

Actions
Copy link
 #4

Updated by szarate about 3 years ago

  • Related to action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process added
Actions
Copy link
 #5

Updated by szarate about 3 years ago

I have left this scrip running on that machine to collect some data at the moment this is happening, use screen -x 75736.virshmonitor_screen to attach to it

should give a rough idea of what's going on at the time this happens (I'm guessing leftover qemu process)

Next step if happens again, enable debug

Actions
Copy link
 #6

Updated by szarate about 3 years ago

Well, we're here and happened again for some other reason: 

Sep 06 11:31:07 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap4': Address already in use

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
    inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
       valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fea6:258a/64 scope link
       valid_lft forever preferred_lft forever
54: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe12:5cd6/64 scope link
       valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febb:c143/64 scope link
       valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe8f:b923/64 scope link
       valid_lft forever preferred_lft forever
58: macvtap4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff

openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 1: 07032013-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_cve@s390x-kvm-sle12

openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12

openQA Instance 5: 07033363-sle-15-SP2-Server-DVD-HA-Incidents-Install-s390x-Build:21035:haproxy-qam-incidentinstall-ha@zkvm

openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12

Sep 06 11:38:26 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap2': Address already in use

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
    inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
       valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fea6:258a/64 scope link
       valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febb:c143/64 scope link
       valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe8f:b923/64 scope link
       valid_lft forever preferred_lft forever
59: macvtap2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff

openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12

openQA Instance 5: 07033373-sle-12-SP4-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-111.1.g7935874-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12

openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12

Sep 06 11:43:15 s390zp19 libvirtd[1684]: Cannot set interface flags on 'macvtap7': Address already in use

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
    inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1488:79ff:fe53:d9ce/64 scope link
       valid_lft forever preferred_lft forever
51: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:a6:25:8a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fea6:258a/64 scope link
       valid_lft forever preferred_lft forever
55: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febb:c143/64 scope link
       valid_lft forever preferred_lft forever
56: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe8f:b923/64 scope link
       valid_lft forever preferred_lft forever
60: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:7b:c2:33 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7b:c233/64 scope link
       valid_lft forever preferred_lft forever
61: macvtap4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:55:e3:d2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe55:e3d2/64 scope link
       valid_lft forever preferred_lft forever
62: macvtap5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:45:3a:6c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe45:3a6c/64 scope link
       valid_lft forever preferred_lft forever
63: macvtap6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe12:5cd6/64 scope link
       valid_lft forever preferred_lft forever
64: macvtap7@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff

openQA Instance 2: 07031707-sle-15-SP2-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 9: 07032012-sle-15-SP3-Server-DVD-Updates-s390x-Build20210906-2-ltp_syscalls@s390x-kvm-sle12

openQA Instance 64: 00006745-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_update@s390x-kvm-sle12

openQA Instance 10: 07033402-sle-15-SP2-Server-DVD-Incidents-Kernel-KOTD-s390x-Build5.3.18-327.1.g9a3c219-ltp_cve_git@s390x-kvm-sle12

openQA Instance 6: 07033393-sle-15-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-148.1.g89b3900-ltp_cve_git@s390x-kvm-sle12

openQA Instance 7: 07033386-sle-15-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-148.1.g89b3900-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12

openQA Instance 1: 07033403-sle-15-SP3-Server-DVD-Incidents-Kernel-KOTD-s390x-Build5.3.18-289.1.gfb75ef9-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12

openQA Instance 5: 07033413-sle-12-SP4-Server-DVD-Incidents-Kernel-KOTD-s390x-Build4.12.14-111.1.g7935874-install_ltp+sle+Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12

openQA Instance 64: 00006737-sle-micro-5.1-MicroOS-Image-s390x-Build57.3_12.13-sle-micro_containers+selinux@s390x-kvm-sle12
Actions
Copy link
 #7

Updated by szarate about 3 years ago

Ok, so the problem is that some mac addresses are being recycled somehow:

52:54:00:8f:b9:23 in this case

BEGIN PROCESSING:   Sep 10 10:37:00 s390zp19 libvirtd[1700]: Cannot set interface flags on 'macvtap7': Address already in use

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 16:88:79:53:d9:ce brd ff:ff:ff:ff:ff:ff
    inet 10.161.159.124/20 brd 10.161.159.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1488:79ff:fe53:d9ce/64 scope link 
       valid_lft forever preferred_lft forever
294: macvtap2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:62:4d:16 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe62:4d16/64 scope link 
       valid_lft forever preferred_lft forever
296: macvtap3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:8f:b9:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe8f:b923/64 scope link 
       valid_lft forever preferred_lft forever
298: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:12:5c:d6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe12:5cd6/64 scope link 
       valid_lft forever preferred_lft forever
299: macvtap4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:7b:c2:33 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7b:c233/64 scope link 
       valid_lft forever preferred_lft forever
300: macvtap6@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:55:e3:d2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe55:e3d2/64 scope link 
       valid_lft forever preferred_lft forever
301: macvtap5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:7f:c6:11 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7f:c611/64 scope link 
       valid_lft forever preferred_lft forever
304: macvtap9@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:45:3a:6c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe45:3a6c/64 scope link 
       valid_lft forever preferred_lft forever
305: macvtap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bb:c1:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febb:c143/64 scope link 
       valid_lft forever preferred_lft forever

openQA Instance 3: 07071743-sle-12-SP5-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests2@s390x-kvm-sle15

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap2    direct   eth0     virtio   52:54:00:62:4d:16


1
openQA Instance 64: 00006848-sle-micro-5.1-DVD-s390x-Build60.1_12.19-sle-micro_installation+update@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap3    direct   eth0     virtio   52:54:00:8f:b9:23


1
openQA Instance 1: 07071585-sle-12-SP4-Server-DVD-Updates-s390x-Build20210910-2-ltp_syscalls@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap0    direct   eth0     virtio   52:54:00:12:5c:d6


1
openQA Instance 10: 07071587-sle-12-SP4-Server-DVD-Updates-s390x-Build20210910-2-ltp_cve@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap4    direct   eth0     virtio   52:54:00:7b:c2:33


1
openQA Instance 6: 07073191-sle-12-SP5-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests1@s390x-kvm-sle15

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap6    direct   eth0     virtio   52:54:00:55:e3:d2


1
openQA Instance 4: 07071966-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-mau-extratests2@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap5    direct   eth0     virtio   52:54:00:7f:c6:11


1
openQA Instance 7: 07071970-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-mau-bind@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap9    direct   eth0     virtio   52:54:00:45:3a:6c


1
openQA Instance 9: 07071976-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-sssd_openldap_functional@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 macvtap1    direct   eth0     virtio   52:54:00:bb:c1:43


1
openQA Instance 5: 07071977-sle-15-SP2-Server-DVD-Updates-s390x-Build20210910-2-sssd_389ds_functional@s390x-kvm-sle12

 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 -           direct   eth0     virtio   52:54:00:8f:b9:23


1
FINISH PROCESSING:  Sep 10 10:37:00 s390zp19 libvirtd[1700]: Cannot set interface flags on 'macvtap7': Address already in use
Actions
Copy link
 #8

Updated by szarate about 3 years ago

  • Related to action #98541: [qe-core][kernel] Steps in case of s390 failures added
Actions
Copy link
 #9

Updated by okurz about 3 years ago

Please consider picking up this ticket within the next 30 days or just set the ticket to the next lower priority of "Normal" (SLO: updated within 365 days).

Actions
Copy link
 #10

Updated by szarate about 3 years ago

  • Status changed from Workable to Resolved
  • Assignee set to szarate

I will call this done, as there's no more errors reported in that case...

I do see 

error: Failed to start domain openQA-SUT-8
error: cannot connect to netlink socket with protocol 0: Address already in use

On this job: https://openqa.suse.de/tests/7332379# but since it happened only once, I'll just ask in discuss-virtualization how that can be improved and Will see later what to do.

Actions
Copy link
 #11

Updated by szarate about 3 years ago

Actions
Copy link
 #12

Updated by szarate about 3 years ago

  • Precedes deleted (action #100841: Improve logging for libvirt hosts)
Actions
Copy link
 #13

Updated by szarate about 3 years ago

Actions
Copy link

Also available in: Atom PDF