QA » openQA Project » openQA Tests

Observation¶

openQA test in scenario sle-15-SP7-Online-x86_64-create_hdd_gnome@svirt-xen-hvm fails in
bootloader_svirt

xen host unreal6 has only 16GB phy memory, and dom0 occupies 3GB. at the same time, we have 8 workers assigned to this host. see https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls

Then memory size for domU is only ~13GB, we may hit not enough memory issue if many vms are allocated 4GB vm for gnome tests.

[2024-09-23T14:40:15.620628Z] [debug] [pid:9354] Using existing SSH connection (key:hostname=unreal6.qe.nue2.suse.org,username=root,port=22)
[2024-09-23T14:40:16.199266Z] [debug] [pid:9354] [run_ssh_cmd(virsh  start openQA-SUT-3 2> >(tee /tmp/os-autoinst-openQA-SUT-3-stderr.log >&2))] stdout:


[2024-09-23T14:40:16.199401Z] [debug] [pid:9354] [run_ssh_cmd(virsh  start openQA-SUT-3 2> >(tee /tmp/os-autoinst-openQA-SUT-3-stderr.log >&2))] stderr:
  error: Failed to start domain 'openQA-SUT-3'
  error: internal error: libxenlight failed to create new domain 'openQA-SUT-3'

libxl-driver.log:19269:2024-10-05 02:07:28.986+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19280:2024-10-05 02:18:19.610+0000: xc: panic: xg_dom_x86.c:1316: meminit_pv: failed to allocate 0x80000 pages: Internal error
libxl-driver.log:19282:2024-10-05 02:18:19.610+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19351:2024-10-05 06:27:36.180+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19355:2024-10-05 06:27:38.275+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19361:2024-10-05 06:27:44.722+0000: xc: panic: xg_dom_x86.c:1316: meminit_pv: failed to allocate 0x100000 pages: Internal error
libxl-driver.log:19363:2024-10-05 06:27:44.722+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19367:2024-10-05 06:27:45.212+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19370:2024-10-05 06:29:19.841+0000: xc: panic: xg_dom_x86.c:1316: meminit_pv: failed to allocate 0x100000 pages: Internal error
libxl-driver.log:19372:2024-10-05 06:29:19.841+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19376:2024-10-05 06:30:09.525+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19380:2024-10-05 06:30:09.653+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19384:2024-10-05 06:30:09.847+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19387:2024-10-05 06:30:09.854+0000: libxl: libxl_create.c:720:libxl__domain_make: domain creation fail: Cannot allocate memory
libxl-driver.log:19390:2024-10-05 06:32:16.124+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19400:2024-10-05 06:48:48.446+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19408:2024-10-05 06:51:05.700+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19446:2024-10-05 08:52:23.451+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19450:2024-10-05 08:52:47.094+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy
libxl-driver.log:19453:2024-10-05 08:52:47.101+0000: libxl: libxl_create.c:720:libxl__domain_make: domain creation fail: Cannot allocate memory

So it could be great if we can increase the memory size to +32GB.

1. Now, we have the other xen host which has 40+GB memory, so we can move the gnome test to that host. I will do via poo#167317
2. If the host has 32GB+ memory, then we can carry out more gnome tests on this host as well

Acceptance criteria¶

• AC1: When All svirt-xen slots on unreal6 run openQA tests Then tests consistently do not fail due to "Cannot allocate memory"

Suggestions¶

• Reduce worker slots to a sustainable number so that memory is not exhausted
• Look into the feasibility of extending the memory

Further details¶

Always latest result in this scenario: latest