action #54863
closed[functional][u] test fails in bootloader_svirt - Missing domains in libvirt but still runnning in XEN.
0%
Description
Observation¶
openQA test in scenario sle-12-SP5-Server-DVD-x86_64-allpatterns@svirt-xen-hvm fails in
bootloader_svirt
Test suite description¶
Maintainers: okurz Installation with all patterns selected for installation to check for potential package conflicts, how the system handles big space usage, etc.
allpatterns installations can take longer, especially on non-x86_64 architectures.
Reproducible¶
Fails since (at least) Build 0251 (current job)
Expected result¶
Last good: 0242 (or more recent)
Further details¶
Always latest result in this scenario: latest
Test died: {¶
'function' => 'define_and_start',
'console' => 'svirt',
'json_cmd_token' => 'RKKSyFeG',
'args' => [],
'cmd' => 'backend_proxy_console_call'
}
virsh start failed at /usr/lib/os-autoinst/consoles/sshVirtsh.pm line 577.
Updated by okurz over 4 years ago
What I can find in the log file:
[2019-07-30T11:43:51.535 CEST] [debug] <<< backend::svirt::run_cmd(Net::SSH2=SCALAR(0x55d6d964a770)='virsh start openQA-SUT-1')
[2019-07-30T11:43:51.782 CEST] [debug] Command's stderr:
error: Failed to start domain openQA-SUT-1
error: internal error: libxenlight failed to create new domain 'openQA-SUT-1'
and then later
[2019-07-30T11:43:52.118 CEST] [debug] Backend process died, backend errors are reported below in the following lines:
Can't call method "map_and_send_key" on an undefined value at /usr/lib/os-autoinst/consoles/vnc_base.pm line 144.
I don't see any recent changes on that method so I assume the error is rather that the object on which this should be called is undefined. Which is most likely obviously the former error causing this. We just ignore that the VM can not be started at all and continue until we fail later.
Updated by szarate over 4 years ago
- Related to action #54758: [tools][u] test fails in bootloader_svirt - Failed to get "write" lock via virsh added
Updated by szarate over 4 years ago
- Is duplicate of action #54173: [virtualization][xen][sporadic] qemu-img Failed to get "write" lock on xen workers (test fails in bootloader_svirt - test fails in bootloader_start) added
Updated by szarate over 4 years ago
- Subject changed from test fails in bootloader_svirt to test fails in bootloader_svirt - Duplicated domains in XEN
- Status changed from Rejected to In Progress
- Assignee set to szarate
Logs at: /var/log/libvirt/libxl/libxl-driver.log reflect the real problem
2019-08-08 12:35:49.511+0000: libxl: libxl_domain.c:81:libxl__domain_rename: Domain 457:Domain with name "openQA-SUT-1" already exists.
2019-08-08 12:35:49.511+0000: libxl: libxl_create.c:961:initiate_domain_create: Domain 457:cannot make domain: -6
2019-08-08 12:35:49.513+0000: libxl: libxl_xshelp.c:201:libxl__xs_read_mandatory: xenstore read failed: `/libxl/457/type': No such file or directory
2019-08-08 12:35:49.513+0000: libxl: libxl_dom.c:52:libxl__domain_type: unable to get domain type for domid=457, assuming HVM
While virsh only shows two domains Dom0 and openQA-hyperv_bridge
talking to xen directly:
# xen-list
Name ID Mem VCPUs State Time(s)
openQA_hyperv_intermediary 1 4088 2 -b--- 27279.5
openQA-SUT-1 153 1024 1 r---- 1094937.8
xen-destroy openQA-SUT-1 works pretty fine :)
Updated by szarate over 4 years ago
- Related to action #54866: [functional][u] test fails in bootloader_svirt added
Updated by SLindoMansilla over 4 years ago
- Subject changed from test fails in bootloader_svirt - Duplicated domains in XEN to [functional][u] test fails in bootloader_svirt - Duplicated domains in XEN
Waiting for feedback from who?
Updated by szarate over 4 years ago
- Status changed from Feedback to In Progress
Waiting for feedback from who?
Myself :), currently writing smalls scripts to guard from this kind of situations, so progress should be the correct status now :)
Updated by szarate over 4 years ago
- Subject changed from [functional][u] test fails in bootloader_svirt - Duplicated domains in XEN to [functional][u] test fails in bootloader_svirt - Missing domains in libvirt but still runnning in XEN.
The followng script has been deployed to openqaw5-xen: https://github.com/foursixnine/stunning-octo-chainsaw/commit/f0fb1084efb42b4104a9a5f6ab7b5689166bccaa
*/5 * * * * /usr/local/bin/clean-xen-domains
Later on, as a follow up to poo#54173 it should be moved to a different repo, and added to salt.
Updated by szarate over 4 years ago
- Status changed from In Progress to Resolved
Updated by szarate about 3 years ago
- Related to action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process added