Project

General

Profile

action #54863

[functional][u] test fails in bootloader_svirt - Missing domains in libvirt but still runnning in XEN.

Added by dheidler over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
-
Start date:
2019-07-30
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-12-SP5-Server-DVD-x86_64-allpatterns@svirt-xen-hvm fails in
bootloader_svirt

Test suite description

Maintainers: okurz Installation with all patterns selected for installation to check for potential package conflicts, how the system handles big space usage, etc.

allpatterns installations can take longer, especially on non-x86_64 architectures.

Reproducible

Fails since (at least) Build 0251 (current job)

Expected result

Last good: 0242 (or more recent)

Further details

Always latest result in this scenario: latest

Test died: {

'function' => 'define_and_start',
'console' => 'svirt',
'json_cmd_token' => 'RKKSyFeG',
'args' => [],
'cmd' => 'backend_proxy_console_call'
}
virsh start failed at /usr/lib/os-autoinst/consoles/sshVirtsh.pm line 577.


Related issues

Related to openQA Infrastructure - action #54758: [tools][u] test fails in bootloader_svirt - Failed to get "write" lock via virsh Rejected2019-07-29

Related to openQA Tests - action #54866: [functional][u] test fails in bootloader_svirtRejected2019-07-30

Related to openQA Tests - action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu processResolved2021-01-26

Is duplicate of openQA Tests - action #54173: [virtualization][xen][sporadic] qemu-img Failed to get "write" lock on xen workers (test fails in bootloader_svirt - test fails in bootloader_start)Closed2019-07-12

History

#1 Updated by okurz over 2 years ago

What I can find in the log file:

[2019-07-30T11:43:51.535 CEST] [debug] <<< backend::svirt::run_cmd(Net::SSH2=SCALAR(0x55d6d964a770)='virsh  start openQA-SUT-1')
[2019-07-30T11:43:51.782 CEST] [debug] Command's stderr:
error: Failed to start domain openQA-SUT-1
error: internal error: libxenlight failed to create new domain 'openQA-SUT-1'

and then later

[2019-07-30T11:43:52.118 CEST] [debug] Backend process died, backend errors are reported below in the following lines:
Can't call method "map_and_send_key" on an undefined value at /usr/lib/os-autoinst/consoles/vnc_base.pm line 144.

I don't see any recent changes on that method so I assume the error is rather that the object on which this should be called is undefined. Which is most likely obviously the former error causing this. We just ignore that the VM can not be started at all and continue until we fail later.

#2 Updated by szarate over 2 years ago

  • Related to action #54758: [tools][u] test fails in bootloader_svirt - Failed to get "write" lock via virsh added

#3 Updated by szarate over 2 years ago

  • Is duplicate of action #54173: [virtualization][xen][sporadic] qemu-img Failed to get "write" lock on xen workers (test fails in bootloader_svirt - test fails in bootloader_start) added

#4 Updated by szarate over 2 years ago

  • Status changed from New to Rejected

see #54173

#5 Updated by szarate over 2 years ago

  • Subject changed from test fails in bootloader_svirt to test fails in bootloader_svirt - Duplicated domains in XEN
  • Status changed from Rejected to In Progress
  • Assignee set to szarate

Logs at: /var/log/libvirt/libxl/libxl-driver.log reflect the real problem

2019-08-08 12:35:49.511+0000: libxl: libxl_domain.c:81:libxl__domain_rename: Domain 457:Domain with name "openQA-SUT-1" already exists.
2019-08-08 12:35:49.511+0000: libxl: libxl_create.c:961:initiate_domain_create: Domain 457:cannot make domain: -6
2019-08-08 12:35:49.513+0000: libxl: libxl_xshelp.c:201:libxl__xs_read_mandatory: xenstore read failed: `/libxl/457/type': No such file or directory
2019-08-08 12:35:49.513+0000: libxl: libxl_dom.c:52:libxl__domain_type: unable to get domain type for domid=457, assuming HVM

While virsh only shows two domains Dom0 and openQA-hyperv_bridge

talking to xen directly:

# xen-list 
Name                                        ID   Mem VCPUs      State   Time(s) 
openQA_hyperv_intermediary                   1  4088     2      -b---   27279.5
openQA-SUT-1                               153  1024     1      r----  1094937.8

xen-destroy openQA-SUT-1 works pretty fine :)

#6 Updated by szarate over 2 years ago

  • Related to action #54866: [functional][u] test fails in bootloader_svirt added

#7 Updated by szarate over 2 years ago

  • Status changed from In Progress to Feedback

#8 Updated by SLindoMansilla over 2 years ago

  • Subject changed from test fails in bootloader_svirt - Duplicated domains in XEN to [functional][u] test fails in bootloader_svirt - Duplicated domains in XEN

Waiting for feedback from who?

#9 Updated by szarate over 2 years ago

  • Status changed from Feedback to In Progress

Waiting for feedback from who?

Myself :), currently writing smalls scripts to guard from this kind of situations, so progress should be the correct status now :)

#10 Updated by szarate over 2 years ago

  • Subject changed from [functional][u] test fails in bootloader_svirt - Duplicated domains in XEN to [functional][u] test fails in bootloader_svirt - Missing domains in libvirt but still runnning in XEN.

The followng script has been deployed to openqaw5-xen: https://github.com/foursixnine/stunning-octo-chainsaw/commit/f0fb1084efb42b4104a9a5f6ab7b5689166bccaa

*/5 * * * * /usr/local/bin/clean-xen-domains

Later on, as a follow up to poo#54173 it should be moved to a different repo, and added to salt.

#11 Updated by szarate over 2 years ago

  • Status changed from In Progress to Resolved

#12 Updated by szarate 12 months ago

  • Related to action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process added

Also available in: Atom PDF