QA &raquo; openQA Project &raquo; openQA Tests

Category:

Bugs in existing tests

Target version:

Start date:

2019-07-30

Due date:

% Done:

Estimated time:

Difficulty:

Description

Observation¶

openQA test in scenario sle-12-SP5-Server-DVD-x86_64-allpatterns@svirt-xen-hvm fails in
bootloader_svirt

Test suite description¶

Maintainers: okurz Installation with all patterns selected for installation to check for potential package conflicts, how the system handles big space usage, etc.

allpatterns installations can take longer, especially on non-x86_64 architectures.

Reproducible¶

Fails since (at least) Build 0251 (current job)

Expected result¶

Last good: 0242 (or more recent)

Further details¶

Always latest result in this scenario: latest

Test died: {¶

'function' => 'define_and_start',
'console' => 'svirt',
'json_cmd_token' => 'RKKSyFeG',
'args' => [],
'cmd' => 'backend_proxy_console_call'
}
virsh start failed at /usr/lib/os-autoinst/consoles/sshVirtsh.pm line 577.

Related issues 4 (0 open — 4 closed)

Related to openQA Infrastructure - action #54758: [tools][u] test fails in bootloader_svirt - Failed to get "write" lock via virsh

Rejected

2019-07-29

Related to openQA Tests - action #54866: [functional][u] test fails in bootloader_svirt

Rejected

2019-07-30

Related to openQA Tests - action #88217: [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process

Resolved

2021-01-26

Is duplicate of openQA Tests - action #54173: [virtualization][xen][sporadic] qemu-img Failed to get "write" lock on xen workers (test fails in bootloader_svirt - test fails in bootloader_start)

Closed

2019-07-12

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Updated by okurz almost 5 years ago

What I can find in the log file:

[2019-07-30T11:43:51.535 CEST] [debug] <<< backend::svirt::run_cmd(Net::SSH2=SCALAR(0x55d6d964a770)='virsh  start openQA-SUT-1')
[2019-07-30T11:43:51.782 CEST] [debug] Command's stderr:
error: Failed to start domain openQA-SUT-1
error: internal error: libxenlight failed to create new domain 'openQA-SUT-1'

and then later

[2019-07-30T11:43:52.118 CEST] [debug] Backend process died, backend errors are reported below in the following lines:
Can't call method "map_and_send_key" on an undefined value at /usr/lib/os-autoinst/consoles/vnc_base.pm line 144.

I don't see any recent changes on that method so I assume the error is rather that the object on which this should be called is undefined. Which is most likely obviously the former error causing this. We just ignore that the VM can not be started at all and continue until we fail later.

Actions

Updated by szarate almost 5 years ago

Related to action #54758: [tools][u] test fails in bootloader_svirt - Failed to get "write" lock via virsh added

Actions

Updated by szarate almost 5 years ago

Is duplicate of action #54173: [virtualization][xen][sporadic] qemu-img Failed to get "write" lock on xen workers (test fails in bootloader_svirt - test fails in bootloader_start) added

Actions

Updated by szarate almost 5 years ago

Status changed from New to Rejected

see #54173

Actions

Updated by szarate over 4 years ago

Subject changed from test fails in bootloader_svirt to test fails in bootloader_svirt - Duplicated domains in XEN
Status changed from Rejected to In Progress
Assignee set to szarate

Logs at: /var/log/libvirt/libxl/libxl-driver.log reflect the real problem

2019-08-08 12:35:49.511+0000: libxl: libxl_domain.c:81:libxl__domain_rename: Domain 457:Domain with name "openQA-SUT-1" already exists.
2019-08-08 12:35:49.511+0000: libxl: libxl_create.c:961:initiate_domain_create: Domain 457:cannot make domain: -6
2019-08-08 12:35:49.513+0000: libxl: libxl_xshelp.c:201:libxl__xs_read_mandatory: xenstore read failed: `/libxl/457/type': No such file or directory
2019-08-08 12:35:49.513+0000: libxl: libxl_dom.c:52:libxl__domain_type: unable to get domain type for domid=457, assuming HVM

While virsh only shows two domains Dom0 and openQA-hyperv_bridge

talking to xen directly:

# xen-list 
Name                                        ID   Mem VCPUs      State   Time(s) 
openQA_hyperv_intermediary                   1  4088     2      -b---   27279.5
openQA-SUT-1                               153  1024     1      r----  1094937.8

xen-destroy openQA-SUT-1 works pretty fine :)

Actions

Updated by szarate over 4 years ago

Related to action #54866: [functional][u] test fails in bootloader_svirt added

Actions

Updated by szarate over 4 years ago

Status changed from In Progress to Feedback

Actions

Updated by SLindoMansilla over 4 years ago

Subject changed from test fails in bootloader_svirt - Duplicated domains in XEN to [functional][u] test fails in bootloader_svirt - Duplicated domains in XEN

Waiting for feedback from who?

Actions

Updated by szarate over 4 years ago

Status changed from Feedback to In Progress

Waiting for feedback from who?

Myself :), currently writing smalls scripts to guard from this kind of situations, so progress should be the correct status now :)

Actions

#10

Updated by szarate over 4 years ago

Subject changed from [functional][u] test fails in bootloader_svirt - Duplicated domains in XEN to [functional][u] test fails in bootloader_svirt - Missing domains in libvirt but still runnning in XEN.

The followng script has been deployed to openqaw5-xen: https://github.com/foursixnine/stunning-octo-chainsaw/commit/f0fb1084efb42b4104a9a5f6ab7b5689166bccaa

*/5 * * * * /usr/local/bin/clean-xen-domains

Later on, as a follow up to poo#54173 it should be moved to a different repo, and added to salt.

Actions

#11

Updated by szarate over 4 years ago

Status changed from In Progress to Resolved

Actions