Project

General

Profile

action #67870

[u] test fails in bootloader_svirt - virsh define failed for domain: openqa-SUT-2

Added by szarate almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Infrastructure
Target version:
-
Start date:
2020-06-09
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-15-SP2-Online-x86_64-allpatterns@svirt-xen-hvm fails in
bootloader_svirt

Test suite description

Maintainers: okurz Installation with all patterns selected for installation to check for potential package conflicts, how the system handles big space usage, etc.

allpatterns installations can take longer, especially on non-x86_64 architectures.

Reproducible

Fails since (at least) Build 207.1 (current job)

Expected result

Last good: 205.1 (or more recent)

Further details

Always latest result in this scenario: latest

libxl-driver.log (207 KB) libxl-driver.log szarate, 2020-06-09 08:40

History

#1 Updated by szarate almost 2 years ago

https://gitlab.suse.de/snippets/658

2020-06-09 08:34:02.833+0000: libxl: libxl_device.c:398:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=qdisk
2020-06-09 08:34:02.833+0000: libxl: libxl_device.c:347:disk_try_backend: Disk vdev=xvda, uses script=(null) on qdisk backend
2020-06-09 08:34:02.833+0000: libxl: libxl_device.c:398:libxl__device_disk_set_backend: Disk vdev=xvdb spec.backend=qdisk
2020-06-09 08:34:02.833+0000: libxl: libxl_device.c:347:disk_try_backend: Disk vdev=xvdb, uses script=(null) on qdisk backend
2020-06-09 08:34:02.833+0000: libxl: libxl_create.c:1017:initiate_domain_create: Domain 1129:running bootloader
2020-06-09 08:34:02.833+0000: libxl: libxl_bootloader.c:335:libxl__bootloader_run: Domain 1129:no bootloader configured, using user supplied kernel
2020-06-09 08:34:02.833+0000: libxl: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x7f1a600029a0: deregister unregistered
2020-06-09 08:34:02.835+0000: libxl: libxl_numa.c:502:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=16, nr_vcpus=36, free_memkb=2529
2020-06-09 08:34:02.835+0000: libxl: libxl_numa.c:502:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=16, nr_vcpus=35, free_memkb=2605
2020-06-09 08:34:02.836+0000: libxl: libxl_dom.c:204:numa_place_domain: NUMA placement candidate with 1 nodes, 16 cpus and 2605 KB free selected
2020-06-09 08:34:02.836+0000: domainbuilder: xc_dom_allocate: cmdline="", features=""
2020-06-09 08:34:02.836+0000: libxl: libxl_dom.c:784:libxl__build_pv: pv kernel mapped 0 path /usr/lib/grub2/x86_64-xen/grub.xen
2020-06-09 08:34:02.836+0000: domainbuilder: xc_dom_kernel_file: filename="/usr/lib/grub2/x86_64-xen/grub.xen"
2020-06-09 08:34:02.836+0000: domainbuilder: xc_dom_malloc_filemap    : 2165 kB
2020-06-09 08:34:02.836+0000: domainbuilder: xc_dom_boot_xen_init: ver 4.11, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
2020-06-09 08:34:02.836+0000: domainbuilder: xc_dom_parse_image: called
2020-06-09 08:34:02.836+0000: domainbuilder: xc_dom_find_loader: trying multiboot-binary loader ... 
2020-06-09 08:34:02.836+0000: domainbuilder: loader probe failed
2020-06-09 08:34:02.837+0000: domainbuilder: xc_dom_find_loader: trying HVM-generic loader ... 
2020-06-09 08:34:02.837+0000: domainbuilder: loader probe failed
2020-06-09 08:34:02.837+0000: domainbuilder: xc_dom_find_loader: trying Linux bzImage loader ... 
2020-06-09 08:34:02.837+0000: domainbuilder: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
2020-06-09 08:34:02.837+0000: domainbuilder: loader probe failed
2020-06-09 08:34:02.837+0000: domainbuilder: xc_dom_find_loader: trying ELF-generic loader ... 
2020-06-09 08:34:02.837+0000: domainbuilder: loader probe OK
2020-06-09 08:34:02.837+0000: xc: ELF: phdr: paddr=0 memsz=0x41d148
2020-06-09 08:34:02.837+0000: xc: ELF: phdr: paddr=0x41d148 memsz=0x20e538
2020-06-09 08:34:02.837+0000: xc: ELF: memory: 0 -> 0x62b680
2020-06-09 08:34:02.837+0000: xc: ELF: note: GUEST_OS = "GRUB2"
2020-06-09 08:34:02.837+0000: xc: ELF: note: LOADER = "generic"
2020-06-09 08:34:02.837+0000: xc: ELF: note: XEN_VERSION = "xen-3.0"
2020-06-09 08:34:02.837+0000: xc: ELF: note: ENTRY = 0
2020-06-09 08:34:02.837+0000: xc: ELF: note: VIRT_BASE = 0
2020-06-09 08:34:02.837+0000: xc: ELF_PADDR_OFFSET unset, using 0
2020-06-09 08:34:02.837+0000: xc: ELF: addresses:
2020-06-09 08:34:02.837+0000: xc:     virt_base        = 0x0
2020-06-09 08:34:02.837+0000: xc:     elf_paddr_offset = 0x0
2020-06-09 08:34:02.837+0000: xc:     virt_offset      = 0x0
2020-06-09 08:34:02.837+0000: xc:     virt_kstart      = 0x0
2020-06-09 08:34:02.837+0000: xc:     virt_kend        = 0x62b680
2020-06-09 08:34:02.837+0000: xc:     virt_entry       = 0x0
2020-06-09 08:34:02.837+0000: xc:     p2m_base         = 0xffffffffffffffff
2020-06-09 08:34:02.837+0000: domainbuilder: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0x0 -> 0x62b680
2020-06-09 08:34:02.837+0000: domainbuilder: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k each
2020-06-09 08:34:02.837+0000: domainbuilder: xc_dom_mem_init: 0x40000 pages
2020-06-09 08:34:02.837+0000: domainbuilder: xc_dom_boot_mem_init: called
2020-06-09 08:34:02.837+0000: domainbuilder: x86_compat: guest xen-3.0-x86_64, address size 64
2020-06-09 08:34:02.842+0000: domainbuilder: xc_dom_malloc            : 2048 kB
2020-06-09 08:34:02.849+0000: domainbuilder: xc_dom_build_image: called
2020-06-09 08:34:02.850+0000: domainbuilder: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x0+0x62c at 0x7f1a24515000
2020-06-09 08:34:02.853+0000: domainbuilder: xc_dom_alloc_segment:   kernel       : 0x0 -> 0x62c000  (pfn 0x0 + 0x62c pages)
2020-06-09 08:34:02.853+0000: xc: ELF: phdr 0 at 0x7f1a24515000 -> 0x7f1a24522f2f
2020-06-09 08:34:02.853+0000: xc: ELF: phdr 2 at 0x7f1a24932148 -> 0x7f1a24b40680
2020-06-09 08:34:02.856+0000: domainbuilder: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x62c+0x200 at 0x7f1a24315000
2020-06-09 08:34:02.856+0000: domainbuilder: xc_dom_alloc_segment:   phys2mach    : 0x62c000 -> 0x82c000  (pfn 0x62c + 0x200 pages)
2020-06-09 08:34:02.856+0000: domainbuilder: xc_dom_alloc_page   :   start info   : 0x82c000 (pfn 0x82c)
2020-06-09 08:34:02.856+0000: domainbuilder: xc_dom_alloc_page   :   xenstore     : 0x82d000 (pfn 0x82d)
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_alloc_page   :   console      : 0x82e000 (pfn 0x82e)
2020-06-09 08:34:02.857+0000: domainbuilder: count_pgtables: 0x0000007fffffffff/39: 0x0000000000000000 -> 0x0000007fffffffff, 1 table(s)
2020-06-09 08:34:02.857+0000: domainbuilder: count_pgtables: 0x000000003fffffff/30: 0x0000000000000000 -> 0x000000003fffffff, 1 table(s)
2020-06-09 08:34:02.857+0000: domainbuilder: count_pgtables: 0x00000000001fffff/21: 0x0000000000000000 -> 0x0000000000bfffff, 6 table(s)
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x82f+0x9 at 0x7f1a6d67f000
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_alloc_segment:   page tables  : 0x82f000 -> 0x838000  (pfn 0x82f + 0x9 pages)
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_alloc_page   :   boot stack   : 0x838000 (pfn 0x838)
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_build_image  : virt_alloc_end : 0x839000
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_build_image  : virt_pgtab_end : 0xc00000
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_boot_image: called
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_update_guest_p2m: dst 64bit, pages 0x40000
2020-06-09 08:34:02.857+0000: domainbuilder: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x82c+0x1 at 0x7f1a6d75a000
2020-06-09 08:34:02.857+0000: domainbuilder: start_info_x86_64: called
2020-06-09 08:34:02.858+0000: domainbuilder: domain builder memory footprint
2020-06-09 08:34:02.858+0000: domainbuilder:    allocated
2020-06-09 08:34:02.858+0000: domainbuilder:       malloc             : 2100 kB
2020-06-09 08:34:02.858+0000: domainbuilder:       anon mmap          : 0 bytes
2020-06-09 08:34:02.858+0000: domainbuilder:    mapped
2020-06-09 08:34:02.858+0000: domainbuilder:       file mmap          : 2165 kB
2020-06-09 08:34:02.858+0000: domainbuilder:       domU mmap          : 8408 kB
2020-06-09 08:34:02.859+0000: domainbuilder: bootlate_pv: shared_info: pfn 0x0, mfn 0x414b
2020-06-09 08:34:02.859+0000: domainbuilder: shared_info_x86_64: called
2020-06-09 08:34:02.859+0000: domainbuilder: vcpu_x86_64: called
2020-06-09 08:34:02.859+0000: domainbuilder: vcpu_x86_64: cr3: pfn 0x82f mfn 0x1111e2f
2020-06-09 08:34:02.861+0000: domainbuilder: xc_dom_release: called
2020-06-09 08:34:02.879+0000: libxl: libxl_device.c:398:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=qdisk
2020-06-09 08:34:02.880+0000: libxl: libxl_device.c:347:disk_try_backend: Disk vdev=xvda, uses script=(null) on qdisk backend
2020-06-09 08:34:02.913+0000: libxl: libxl_linux.c:226:libxl__get_hotplug_script_info: Domain 1129:backend_kind 3, no need to execute scripts
2020-06-09 08:34:02.914+0000: libxl: libxl_device.c:1204:device_hotplug: Domain 1129:No hotplug script to execute
2020-06-09 08:34:02.914+0000: libxl: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x7f1a60003f60: deregister unregistered
2020-06-09 08:34:02.914+0000: libxl: libxl_device.c:398:libxl__device_disk_set_backend: Disk vdev=xvdb spec.backend=qdisk
2020-06-09 08:34:02.914+0000: libxl: libxl_device.c:347:disk_try_backend: Disk vdev=xvdb, uses script=(null) on qdisk backend
2020-06-09 08:34:02.950+0000: libxl: libxl_linux.c:226:libxl__get_hotplug_script_info: Domain 1129:backend_kind 3, no need to execute scripts
2020-06-09 08:34:02.950+0000: libxl: libxl_device.c:1204:device_hotplug: Domain 1129:No hotplug script to execute
2020-06-09 08:34:02.950+0000: libxl: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x7f1a60006aa0: deregister unregistered
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2372:libxl__spawn_local_dm: Domain 1129:Spawning device-model /usr/bin/qemu-system-i386 with arguments:
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  /usr/bin/qemu-system-i386
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -xen-domid
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  1129
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -chardev
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-1129,server,nowait
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -no-shutdown
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -mon
2020-06-09 08:34:03.002+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  chardev=libxl-cmd,mode=control
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -chardev
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-1129,server,nowait
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -mon
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  chardev=libxenstat-cmd,mode=control
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -nodefaults
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -no-user-config
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -xen-attach
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -name
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  openQA-SUT-22
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -vnc
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  0.0.0.0:2,password
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -display
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  none
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -machine
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  xenpv
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  -m
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2374:libxl__spawn_local_dm: Domain 1129:  1024
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2376:libxl__spawn_local_dm: Domain 1129:Spawning device-model /usr/bin/qemu-system-i386 with additional environment:
2020-06-09 08:34:03.003+0000: libxl: libxl_dm.c:2378:libxl__spawn_local_dm: Domain 1129:  XEN_QEMU_CONSOLE_LIMIT=1048576
2020-06-09 08:34:03.003+0000: libxl: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0x7f1a60002ca8 wpath=/local/domain/0/device-model/1129/state token=2/8: register slotnum=2
2020-06-09 08:34:03.025+0000: libxl: libxl_event.c:573:watchfd_callback: watch w=0x7f1a60002ca8 wpath=/local/domain/0/device-model/1129/state token=2/8: event epath=/local/domain/0/device-model/1129/state
2020-06-09 08:34:03.025+0000: libxl: libxl_exec.c:398:spawn_watch_event: domain 1129 device model: spawn watch p=(null)
2020-06-09 08:34:03.186+0000: libxl: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=0x7f1a60002ca8 wpath=/local/domain/0/device-model/1129/state token=2/8: deregister slotnum=2
2020-06-09 08:34:03.186+0000: libxl: libxl_dm.c:2468:device_model_spawn_outcome: Domain 1129:domain 1129 device model: spawn failed (rc=-3)
2020-06-09 08:34:03.186+0000: libxl: libxl_create.c:1577:domcreate_devmodel_started: Domain 1129:device model did not start: -3
2020-06-09 08:34:03.188+0000: libxl: libxl_dm.c:2582:kill_device_model: Device Model already exited
2020-06-09 08:34:05.193+0000: libxl: libxl_linux.c:226:libxl__get_hotplug_script_info: Domain 1129:backend_kind 3, no need to execute scripts
2020-06-09 08:34:05.193+0000: libxl: libxl_device.c:1204:device_hotplug: Domain 1129:No hotplug script to execute
2020-06-09 08:34:05.193+0000: libxl: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x7f1a6000ce70: deregister unregistered
2020-06-09 08:34:05.216+0000: libxl: libxl_linux.c:226:libxl__get_hotplug_script_info: Domain 1129:backend_kind 3, no need to execute scripts
2020-06-09 08:34:05.216+0000: libxl: libxl_device.c:1204:device_hotplug: Domain 1129:No hotplug script to execute
2020-06-09 08:34:05.216+0000: libxl: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x7f1a6000d190: deregister unregistered
2020-06-09 08:34:05.230+0000: libxl: libxl_linux.c:226:libxl__get_hotplug_script_info: Domain 1129:backend_kind 5, no need to execute scripts
2020-06-09 08:34:05.230+0000: libxl: libxl_device.c:1204:device_hotplug: Domain 1129:No hotplug script to execute
2020-06-09 08:34:05.230+0000: libxl: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x7f1a6000d4b0: deregister unregistered
2020-06-09 08:34:05.242+0000: libxl: libxl_linux.c:226:libxl__get_hotplug_script_info: Domain 1129:backend_kind 6, no need to execute scripts
2020-06-09 08:34:05.242+0000: libxl: libxl_device.c:1204:device_hotplug: Domain 1129:No hotplug script to execute
2020-06-09 08:34:05.242+0000: libxl: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x7f1a60007950: deregister unregistered
2020-06-09 08:34:05.281+0000: libxl: libxl_domain.c:1193:devices_destroy_cb: Domain 1129:Forked pid 3282 for destroy of domain
2020-06-09 08:34:05.344+0000: libxl: libxl_event.c:573:watchfd_callback: watch w=0x7f1a0c0064f0 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
2020-06-09 08:34:05.345+0000: libxl: libxl_domain.c:792:domain_death_xswatch_callback: [evg=0] all reported
2020-06-09 08:34:05.345+0000: libxl: libxl_domain.c:854:domain_death_xswatch_callback: domain death search done
2020-06-09 08:34:05.350+0000: libxl: libxl_event.c:1873:libxl__ao_complete: ao 0x7f1a600048b0: complete, rc=-3
2020-06-09 08:34:05.350+0000: libxl: libxl_event.c:1842:libxl__ao__destroy: ao 0x7f1a600048b0: destroy

#3 Updated by szarate almost 2 years ago

  • Subject changed from test fails in bootloader_svirt - virsh define failed on openqa-SUT-2 to [u] test fails in bootloader_svirt - virsh define failed for domain: openqa-SUT-2

So, running the qemu command, shows:

qemu-system-i386: -vnc 0.0.0.0:2,password: Failed to start VNC server: Failed to find an available port: Address already in use

#4 Updated by szarate almost 2 years ago

so after

ss -nutlp | grep qemu

And killing the leftover process:

tcp   LISTEN     0      1                                                                                  0.0.0.0:5902                                                                                           0.0.0.0:*                   users:(("qemu-system-i38",pid=5121,fd=41))                

vm's can be spawned again

as a bonus changed libvirtd to log libxl activity as suggested by Olaf Hering:

#file:/etc/libvirt/libvirtd.conf
log_level = 1
log_outputs="1:file:/dev/null"

#5 Updated by szarate almost 2 years ago

  • Status changed from In Progress to Resolved
virsh desc openQA-SUT-2
openQA Instance 2: 04330472-sle-15-SP2-Online-x86_64-Build207.1-allpatterns@svirt-xen-pv

https://openqa.suse.de/tests/4330472

Also available in: Atom PDF