Project

General

Profile

Actions

action #134357

open

[tools] kvm test run fails on OW21 and OW22 - but works on OW4

Added by dimstar 9 months ago. Updated 8 months ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
Start date:
2023-08-17
Due date:
% Done:

0%

Estimated time:
Difficulty:
Tags:

Description

Observation

openQA test in scenario microos-Tumbleweed-MicroOS-Image-x86_64-microos_qemu@64bit fails in
kvm

Did not really spot what is going on - but whenever this runs on OW4, it passes, while it fails on OW21/OW22

Test suite description

Reproducible

Fails since (at least) Build 20230816

Expected result

Last good: 20230815 (or more recent)

Further details

Always latest result in this scenario: latest


Related issues 1 (0 open1 closed)

Related to openQA Tests - action #134285: [tools] Openqa bootstrap: test fails in test_results size:M auto_review:"Test died: no candidate needle with tag.+openqa-testresult"Resolvedlivdywan2023-08-15

Actions
Actions #1

Updated by dimstar 9 months ago

  • Related to action #134285: [tools] Openqa bootstrap: test fails in test_results size:M auto_review:"Test died: no candidate needle with tag.+openqa-testresult" added
Actions #2

Updated by favogt 9 months ago

With the qemu monitor and debug logs it's visible that KVM never makes any progress. It tries to run the vcpu but it never runs the first instruction. IP is stuck at FFF0.

I suspect a kernel bug, probably on the (openQA) VM host side. Using the latest TW kernel on the worker didn't help though.

Actions #3

Updated by dimstar 9 months ago

Some experiments hint at a kernel bug on the host: https://bugzilla.opensuse.org/show_bug.cgi?id=1214537

Actions #4

Updated by favogt 9 months ago

  • Status changed from New to Rejected

Confirmed as product bug meanwhile.

I used ow19 for performing some tests (and building test kernels. The workers are nice kernel build hosts!).
We could identify that the latest upstream kernel 6.5-rc7 works (while -rc6 is broken). I'll keep ow19 reserved for a bit still to test a kernel with backported fixes.

Actions #5

Updated by okurz 9 months ago

  • Status changed from Rejected to Blocked
  • Assignee set to okurz
  • Target version set to Ready

Then let's keep this ticket open as tracker until we can again run the latest stable distribution kernel. I will track the bug. Should be downgrade and lock the older kernel version on more hosts? What if we upgrade more workers to Leap 15.5?

Actions #6

Updated by favogt 9 months ago

okurz wrote in #note-5:

Then let's keep this ticket open as tracker until we can again run the latest stable distribution kennel t. I will track the bug. Should be downgrade and lock the older kernel version on more hosts? What if we upgrade more workers to Leap 15.5?

15.4 and 15.5 are equally affected. As a workaround, kernel 6.5-rc7+ can be installed or spec_rstack_overflow=off added to the cmdline.

Actions #7

Updated by openqa_review 9 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: extra_tests_vagrant
https://openqa.opensuse.org/tests/3558849#step/sshfs/1

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions #8

Updated by maritawerner 9 months ago

  • Subject changed from kvm test run fails on OW21 and OW22 - but works on OW4 to [tools] kvm test run fails on OW21 and OW22 - but works on OW4
Actions #9

Updated by okurz 8 months ago

  • Target version changed from Ready to Tools - Next
Actions #10

Updated by dheidler 8 months ago

FTR: Workarounds are currently applied on openqaworker21-26

Working kernel versions for 15.4 and 15.5:

kernel-default-5.14.21-150400.24.74.1.x86_64
kernel-default-5.14.21-150500.55.12.1.x86_64

zypper in --oldpackage kernel-default{,-extra,-optional}-5.14.21-150500.55.12.1.x86_64
zypper al -m bsc#1214537 'kernel*'
Actions

Also available in: Atom PDF