Project

General

Profile

Actions

action #88217

closed

[qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process

Added by SLindoMansilla over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
-
Start date:
2021-01-26
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

error: Failed to start domain openQA-SUT-1
error: internal error: libxenlight failed to create new domain 'openQA-SUT-1'

Reproducible

  • In all xen machines
  • Fails since Build 132.4
  • Current occurrence for scenario [sle-15-SP3-Online-x86_64-Build132.4-memtest@svirt-xen-hvm](bootloader_svirt)
  • Last good: 130.3
  • latest

Related issues 3 (0 open3 closed)

Related to openQA Tests - action #54863: [functional][u] test fails in bootloader_svirt - Missing domains in libvirt but still runnning in XEN.Resolvedszarate2019-07-30

Actions
Related to openQA Infrastructure - action #88299: [virtualization] Worker openqaw5-xen-1.qa.suse.de is not reachable (xen-hvm/xen-pv failing)Resolvedmloviska2021-01-28

Actions
Related to openQA Tests - action #97532: [qe-core][sporadic] s390x jobs are failing to boot auto_review:"error: Cannot set interface flags on 'macvtap.*': Address already in use":retryResolvedszarate

Actions
Actions #1

Updated by SLindoMansilla over 3 years ago

  • Subject changed from [qe-core] test fails in bootloader_svirt to [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain
Actions #2

Updated by szarate over 3 years ago

  • Related to action #54863: [functional][u] test fails in bootloader_svirt - Missing domains in libvirt but still runnning in XEN. added
Actions #3

Updated by szarate over 3 years ago

  • Status changed from New to In Progress
  • Assignee set to szarate

Smells like: poo#54863

Actions #4

Updated by szarate over 3 years ago

  • Subject changed from [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain to [qe-core] test fails in bootloader_svirt - libxenlight failed to create new domain: leftover qemu process
  • Status changed from In Progress to Resolved

I modified the script added before for a similar problem.

What happened here is:

At some point due to possibly a bug in libvirt, when a domain is destroyed, there could be leftovers in xen (as previously mentioned), and sometimes, the leftovers could be left in a worse state, like leaving a qemu process roaming around, pgrep -f qemu.*openQA-SUT- comes particularly handy here.

I remember something similar happening in the past, so for the time being I updated the cleanup script to also kill the qemu process (https://github.com/foursixnine/stunning-octo-chainsaw/commit/be43227d76bfd55bfd99e2311e61d8faa8c8ed36)

For now, https://openqa.suse.de/tests/05347291#live is running on the virsh instance that was previously failing.

Also the cron job has been changed to only run once a day, but I'll be getting emails if failures ocurr (later to be moved to one of the monitoring mailing lists??)

Actions #5

Updated by szarate over 3 years ago

  • Related to action #88299: [virtualization] Worker openqaw5-xen-1.qa.suse.de is not reachable (xen-hvm/xen-pv failing) added
Actions #6

Updated by okurz over 3 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: extra_tests_gnome@svirt-xen-pv
https://openqa.suse.de/tests/5478993

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed
Actions #7

Updated by okurz over 3 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: extra_tests_gnome@svirt-xen-pv
https://openqa.suse.de/tests/5585765

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed
Actions #8

Updated by okurz over 3 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: extra_tests_gnome@svirt-xen-pv
https://openqa.suse.de/tests/5585765

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed
Actions #9

Updated by szarate about 3 years ago

  • Related to action #97532: [qe-core][sporadic] s390x jobs are failing to boot auto_review:"error: Cannot set interface flags on 'macvtap.*': Address already in use":retry added
Actions

Also available in: Atom PDF