action #36754

[functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation

Added by nicksinger over 1 year ago. Updated 10 months ago.

Status:BlockedStart date:04/06/2018
Priority:NormalDue date:
Assignee:SLindoMansilla% Done:

0%

Category:Bugs in existing tests
Target version:QA - future
Difficulty:
Duration:

Description

Observation

Could not access KVM kernel module: No such device

Reproducible

Acceptance criteria

AC1 Test suite suse_patches-systemd_testsuite doesn't fail on missing KVM kernel module

Tasks

  1. Enable KVM virtualization on openQA-worker hosts.
  2. If necessary, install SLE15 for aarch64 openQA-workers hosts (tblume is not sure if we can do it on SLE12)

Expected result

Last good build 563.1:


Related issues

Related to openQA Tests - action #34996: [functional][opensuse][u][epic] test fails in systemd_tes... Blocked 25/06/2018
Related to openQA Project - action #32563: [functional][u] fix salt for power Resolved 28/02/2018
Related to openQA Tests - action #44150: [functional][u] test fails in systemd_testsuite, Test die... Resolved 21/11/2018
Related to openQA Tests - action #25248: [tools][functional][systemd][u] openQA-workers for system... Resolved 13/09/2017
Related to openQA Tests - action #45158: [systemd] Implement systemd testsuite as openQA perl module In Progress 18/01/2019
Duplicated by openQA Tests - action #43826: [qam] [SLE15] test fails in systemd_testsuite Rejected 15/11/2018
Blocked by openQA Tests - action #33202: [sle][functional][s390x][zkvm][u][hard] test fails in boo... Resolved 13/03/2018 14/08/2018
Blocked by openQA Tests - action #44468: [functional][u][labs] Proper handling of assets for svirt... Feedback 28/11/2018

History

#1 Updated by SLindoMansilla over 1 year ago

  • Related to action #34996: [functional][opensuse][u][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added

#2 Updated by okurz over 1 year ago

  • Subject changed from [functional][systemd][u] test fails in systemd_testsuite on aarch64 - needs further investigation to [functional][systemd][u][fast] test fails in systemd_testsuite on aarch64 - needs further investigation
  • Due date set to 19/06/2018
  • Priority changed from Normal to Urgent
  • Target version set to Milestone 17

happening in SLE15 GMC.

#3 Updated by mgriessmeier over 1 year ago

  • Subject changed from [functional][systemd][u][fast] test fails in systemd_testsuite on aarch64 - needs further investigation to [functional][systemd][u][fast][medium] test fails in systemd_testsuite - needs further investigation
  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by SLindoMansilla over 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to SLindoMansilla

#5 Updated by SLindoMansilla over 1 year ago

First SR to reorganize SUSE patches accepted: https://build.opensuse.org/request/show/615165

#6 Updated by SLindoMansilla over 1 year ago

aarch64 issue is a bug and is handle in https://bugzilla.suse.com/show_bug.cgi?id=1097440

#7 Updated by SLindoMansilla over 1 year ago

  • Description updated (diff)

SR to patch upstream bug: https://build.opensuse.org/request/show/616259
SR to patch openSUSE bug: https://build.opensuse.org/request/show/616468

Investigating s390x issue.

#8 Updated by SLindoMansilla over 1 year ago

  • Status changed from In Progress to Blocked
  • Priority changed from Urgent to High

At the moment SLE15 is already on GMC phase. Adjusting prio.

I cannot investigate further due to a malfunction of the s390x-susekvm worker: https://progress.opensuse.org/issues/33202

#9 Updated by SLindoMansilla over 1 year ago

  • Blocked by action #33202: [sle][functional][s390x][zkvm][u][hard] test fails in boot_to_desktop - still insufficient error reporting, black screen with mouse cursor - we all hate it (was: I hate it) added

#10 Updated by SLindoMansilla over 1 year ago

  • Status changed from Blocked to In Progress

I forgot to change status after s390x worker was self-healed

#11 Updated by SLindoMansilla over 1 year ago

Follow up for "ID_LIKE" patch to be fully back ported into SLE15: https://github.com/openSUSE/systemd/blob/SLE15/test/test-functions

#12 Updated by okurz over 1 year ago

  • Target version changed from Milestone 17 to Milestone 17

#13 Updated by SLindoMansilla over 1 year ago

  • Description updated (diff)
  • Priority changed from High to Normal

The problem is identified.

SLE15 is GM, so lowering priority.

#14 Updated by SLindoMansilla over 1 year ago

  • Description updated (diff)
  • Assignee changed from SLindoMansilla to tsaupe

#15 Updated by SLindoMansilla over 1 year ago

  • Blocked by action #25248: [tools][functional][systemd][u] openQA-workers for systemd department added

#16 Updated by mgriessmeier over 1 year ago

  • Due date changed from 19/06/2018 to 03/07/2018

#17 Updated by okurz over 1 year ago

  • Subject changed from [functional][systemd][u][fast][medium] test fails in systemd_testsuite - needs further investigation to [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation
  • Due date deleted (03/07/2018)
  • Target version changed from Milestone 17 to Milestone 20

So I discussed in detail with slindomansilla related to #34996 as well. AFAIU this ticket is for SLE in particular. Currently we focus on SLE12SP4 and for SLE15 the tests had been failing for a long time. This is why I delay the ticket plan to M20 as we should get openSUSE Tumbleweed in a stable state first and then revisit the situation for SLE tests. Whenever we see tests failing consistently and for a long time we should move the test scenario to the according "test development" job groups on the openQA server(s). Let's focus into #34996 . @tsaupe of course the plan is only according to our planning as QSF (QA SLE functional) to look into this issue again later, milestone 20, which is 2018-11, but you are free to continue on this ticket as you like :)

#18 Updated by okurz over 1 year ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

#19 Updated by okurz over 1 year ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

#20 Updated by okurz over 1 year ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

#21 Updated by okurz over 1 year ago

  • Status changed from In Progress to Workable
  • Assignee deleted (tsaupe)
  • Priority changed from Normal to High

Seems like no real progress here. By now I think we should take a look into SLE15 tests, at least remove the red blob from the validation tests. https://bugzilla.suse.com/show_bug.cgi?id=1105101 is currently used as label in these tests but also there nothing is moving forward, multiple unattended openqa-review reminder comments.

#22 Updated by nicksinger over 1 year ago

  • Related to action #32563: [functional][u] fix salt for power added

#23 Updated by SLindoMansilla over 1 year ago

  • Status changed from Workable to Feedback
  • Assignee set to SLindoMansilla

A SR provided by tsaupe was accepted yesterday: https://build.suse.de/request/show/175568

I will verify the fix on OSD.
Waiting for:

#24 Updated by SLindoMansilla over 1 year ago

  • Status changed from Feedback to In Progress

The job times out. I will try to reproduce how much time it needs to adapt the test.

#25 Updated by SLindoMansilla over 1 year ago

  • Status changed from In Progress to Feedback

#27 Updated by SLindoMansilla over 1 year ago

  • Status changed from Feedback to In Progress

Both fail. 1 hour timeout is not enough. Investigating...

#28 Updated by SLindoMansilla over 1 year ago

tblume has prepared a SR to solve this issue that makes the test hang in the qemu VM: https://build.suse.de/request/show/177535

#29 Updated by SLindoMansilla over 1 year ago

SR is accepted, waiting for the package to build on QA:SLE15: https://build.suse.de/package/show/QA:SLE15/systemd-v234-testsuite

#30 Updated by SLindoMansilla over 1 year ago

  • Status changed from In Progress to Feedback

Waiting for verification on OSD:

#31 Updated by SLindoMansilla over 1 year ago

It is still hanging on second test. Waiting for feedback from tblume.

#32 Updated by SLindoMansilla over 1 year ago

  • Duplicated by action #43826: [qam] [SLE15] test fails in systemd_testsuite added

#33 Updated by SLindoMansilla over 1 year ago

Last SR broke also x86_64 tests: https://openqa.suse.de/tests/2259623#step/systemd_testsuite/13

Reverting last SR

#34 Updated by SLindoMansilla over 1 year ago

tblume already provided a new SR with the fix: https://build.suse.de/request/show/177653

Not needed to revert changes.

#35 Updated by SLindoMansilla over 1 year ago

Waiting for verification on OSD:

#36 Updated by SLindoMansilla over 1 year ago

  • Status changed from Feedback to In Progress

Results for build 96.7:

Investigating fail on TEST-10:

#38 Updated by SLindoMansilla over 1 year ago

  • Status changed from In Progress to Feedback

New SR from tblume to fix aarch64 problem: https://build.suse.de/request/show/177815

Waiting for verification: https://openqa.suse.de/tests/2263888

#39 Updated by SLindoMansilla over 1 year ago

TEST-02 passes, but it fails on other tests later.

Verifying that the SR is not breaking s390x and x86_64 scenarios:

#40 Updated by okurz over 1 year ago

  • Related to action #44150: [functional][u] test fails in systemd_testsuite, Test died: Could not retrieve required variable QA_HEAD_REPO at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/console/systemd_testsuite.pm line 21. added

#41 Updated by okurz over 1 year ago

  • Priority changed from High to Urgent
  • Target version changed from Milestone 20 to Milestone 21

M20 is over, as we already shifted a lot I am not sure how this will go on. As we still have jobs failing labeled with this ticket, e.g. https://openqa.suse.de/tests/2271738#step/systemd_testsuite/17, I see it as urgent now. The urgency could be handled by removing the scenario again. But I leave it to you to decide which way to go, just … urgently please :)

#42 Updated by SLindoMansilla about 1 year ago

  • Status changed from Feedback to In Progress

At the moment there is no s390x worker to verify.
OSD ones are getting freeze.
QSF shared workers are not working because the s390x zkvm backend was never adapted to work on workers with enabled cache.

Fixing s390x backend first and moving aarch64 and ppc64le scenarios to development job group: https://openqa.suse.de/admin/job_templates/96

#43 Updated by SLindoMansilla about 1 year ago

  • Priority changed from Urgent to High

Urgency removed after moving aarch64 and ppc64le scenarios to development job group.

@okurz, feel free to set normal prio.

#44 Updated by okurz about 1 year ago

  • Priority changed from High to Normal

yes, normal prio is fine.

#45 Updated by SLindoMansilla about 1 year ago

  • Status changed from In Progress to Blocked

#46 Updated by SLindoMansilla about 1 year ago

  • Status changed from Blocked to In Progress

#47 Updated by SLindoMansilla about 1 year ago

Generating the missing qcow2 image: http://slindomansilla-vm.qa.suse.de/tests/962

#48 Updated by SLindoMansilla about 1 year ago

  • Related to action #44468: [functional][u][labs] Proper handling of assets for svirt workers added

#49 Updated by SLindoMansilla about 1 year ago

  • Related to deleted (action #44468: [functional][u][labs] Proper handling of assets for svirt workers)

#50 Updated by SLindoMansilla about 1 year ago

  • Blocked by action #44468: [functional][u][labs] Proper handling of assets for svirt workers added

#51 Updated by SLindoMansilla about 1 year ago

  • Blocked by deleted (action #25248: [tools][functional][systemd][u] openQA-workers for systemd department)

#52 Updated by SLindoMansilla about 1 year ago

  • Related to action #25248: [tools][functional][systemd][u] openQA-workers for systemd department added

#53 Updated by SLindoMansilla about 1 year ago

  • Status changed from In Progress to Feedback

s390x verification failed: https://openqa.suse.de/tests/2288353

Waiting for tsaupe's feedback on https://build.suse.de/request/show/177956

#54 Updated by SLindoMansilla about 1 year ago

  • Status changed from Feedback to Workable

Not able to verify on s390x until there is an available qcow2 image of SLE15-SP1 textmode for s390x-kvm-sle12

#55 Updated by SLindoMansilla about 1 year ago

  • Related to action #45158: [systemd] Implement systemd testsuite as openQA perl module added

#56 Updated by SLindoMansilla about 1 year ago

  • Status changed from Workable to Blocked

Blocked by #45158

#57 Updated by SLindoMansilla about 1 year ago

  • Related to deleted (action #45158: [systemd] Implement systemd testsuite as openQA perl module)

#58 Updated by SLindoMansilla about 1 year ago

  • Blocked by action #45158: [systemd] Implement systemd testsuite as openQA perl module added

#59 Updated by okurz about 1 year ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2343909

#60 Updated by okurz about 1 year ago

  • Target version changed from Milestone 21 to future

due to blocker

#61 Updated by okurz about 1 year ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2429581

#62 Updated by okurz about 1 year ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2452703

#63 Updated by SLindoMansilla about 1 year ago

The test suite was moved to the development job group due to the bug in existing tests that are taking long time to be fixed: https://openqa.suse.de/admin/job_templates/96

Latest: https://openqa.suse.de/tests/latest?version=15-SP1&distri=sle&test=suse_patches-systemd_testsuite&machine=64bit&arch=x86_64&flavor=Installer-DVD#next_previous

#65 Updated by SLindoMansilla 10 months ago

  • Blocked by deleted (action #45158: [systemd] Implement systemd testsuite as openQA perl module)

#66 Updated by SLindoMansilla 10 months ago

  • Related to action #45158: [systemd] Implement systemd testsuite as openQA perl module added

Also available in: Atom PDF