action #167956
Updated by okurz about 1 month ago
## Observation Description openQA test in scenario sle-15-SP7-Online-x86_64-create_hdd_gnome@svirt-xen-hvm fails in [bootloader_svirt](https://openqa.suse.de/tests/15403428/modules/bootloader_svirt/steps/38) xen host `unreal6` has only 16GB phy memory, and dom0 occupies 3GB. at the same time, we have 8 workers assigned to this host. see https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls Then memory size for domU is only ~13GB, we may hit not enough memory issue if many vms are allocated 4GB vm for gnome tests. ``` [2024-09-23T14:40:15.620628Z] [debug] [pid:9354] Using existing SSH connection (key:hostname=unreal6.qe.nue2.suse.org,username=root,port=22) [2024-09-23T14:40:16.199266Z] [debug] [pid:9354] [run_ssh_cmd(virsh start openQA-SUT-3 2> >(tee /tmp/os-autoinst-openQA-SUT-3-stderr.log >&2))] stdout: [2024-09-23T14:40:16.199401Z] [debug] [pid:9354] [run_ssh_cmd(virsh start openQA-SUT-3 2> >(tee /tmp/os-autoinst-openQA-SUT-3-stderr.log >&2))] stderr: error: Failed to start domain 'openQA-SUT-3' error: internal error: libxenlight failed to create new domain 'openQA-SUT-3' libxl-driver.log:19269:2024-10-05 02:07:28.986+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19280:2024-10-05 02:18:19.610+0000: xc: panic: xg_dom_x86.c:1316: meminit_pv: failed to allocate 0x80000 pages: Internal error libxl-driver.log:19282:2024-10-05 02:18:19.610+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19351:2024-10-05 06:27:36.180+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19355:2024-10-05 06:27:38.275+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19361:2024-10-05 06:27:44.722+0000: xc: panic: xg_dom_x86.c:1316: meminit_pv: failed to allocate 0x100000 pages: Internal error libxl-driver.log:19363:2024-10-05 06:27:44.722+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19367:2024-10-05 06:27:45.212+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19370:2024-10-05 06:29:19.841+0000: xc: panic: xg_dom_x86.c:1316: meminit_pv: failed to allocate 0x100000 pages: Internal error libxl-driver.log:19372:2024-10-05 06:29:19.841+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19376:2024-10-05 06:30:09.525+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19380:2024-10-05 06:30:09.653+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19384:2024-10-05 06:30:09.847+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19387:2024-10-05 06:30:09.854+0000: libxl: libxl_create.c:720:libxl__domain_make: domain creation fail: Cannot allocate memory libxl-driver.log:19390:2024-10-05 06:32:16.124+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19400:2024-10-05 06:48:48.446+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19408:2024-10-05 06:51:05.700+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19446:2024-10-05 08:52:23.451+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19450:2024-10-05 08:52:47.094+0000: libxl: libxl_dom.c:581:libxl__build_dom: xc_dom_boot_mem_init failed: Device or resource busy libxl-driver.log:19453:2024-10-05 08:52:47.101+0000: libxl: libxl_create.c:720:libxl__domain_make: domain creation fail: Cannot allocate memory ``` So it could be great if we can increase the memory size to +32GB. ## Comments 1. Now, we have the other xen host which has 40+GB memory, so we can move the gnome test to that host. I will do via poo#167317 2. If the host has 32GB+ memory, then we can carry out more gnome tests on this host as well ## Acceptance criteria Observation openQA test in scenario sle-15-SP7-Online-x86_64-create_hdd_gnome@svirt-xen-hvm fails in * **AC1:** *When* All svirt-xen slots on unreal6 run openQA tests *Then* tests consistently do not fail due to "Cannot allocate memory" [bootloader_svirt](https://openqa.suse.de/tests/15403428/modules/bootloader_svirt/steps/38) ## Suggestions Test suite description * Reduce worker slots to a sustainable number so that memory is not exhausted * Look into the feasibility of extending the memory image creation job used as parent for other jobs testing based on existing installation. To be used as `START_AFTER_TEST=create_hdd_gnome` ## Reproducible Fails since (at least) Build [18.1](https://openqa.suse.de/tests/15403428) (current job) ## Expected result Last good: [14.1](https://openqa.suse.de/tests/15353330) (or more recent) ## Further details Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=svirt-xen-hvm&test=create_hdd_gnome&version=15-SP7)