Project

General

Profile

action #36754

[qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation

Added by nicksinger about 5 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
SLindoMansilla
Category:
Bugs in existing tests
Target version:
QA - future
Start date:
2018-06-04
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

Could not access KVM kernel module: No such device

Reproducible

Acceptance criteria

AC1 Test suite suse_patches-systemd_testsuite doesn't fail on missing KVM kernel module

Tasks

  1. Enable KVM virtualization on openQA-worker hosts.
  2. If necessary, install SLE15 for aarch64 openQA-workers hosts (tblume is not sure if we can do it on SLE12)

Expected result

Last good build 563.1:

Related issues

Related to openQA Tests - coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfileRejected2018-06-25

Related to openQA Project - action #32563: [functional][u] fix salt for powerResolved2018-02-28

Related to openQA Tests - action #44150: [functional][u] test fails in systemd_testsuite, Test died: Could not retrieve required variable QA_HEAD_REPO at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/console/systemd_testsuite.pm line 21.Resolved2018-11-21

Related to openQA Tests - action #25248: [tools][functional][systemd][u] openQA-workers for systemd departmentResolved2017-09-13

Related to openQA Tests - action #45158: [systemd] Implement systemd testsuite as openQA perl moduleResolved2019-01-18

Related to openQA Tests - action #44468: [qe-core][functional][tools] Proper handling of assets for svirt workersWorkable2018-11-28

Has duplicate openQA Tests - action #43826: [qam] [SLE15] test fails in systemd_testsuiteRejected2018-11-15

Blocked by openQA Tests - action #33202: [sle][functional][s390x][zkvm][u][hard] test fails in boot_to_desktop - still insufficient error reporting, black screen with mouse cursor - we all hate it (was: I hate it)Resolved2018-03-132018-08-14

History

#1 Updated by SLindoMansilla about 5 years ago

  • Related to coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added

#2 Updated by okurz about 5 years ago

  • Subject changed from [functional][systemd][u] test fails in systemd_testsuite on aarch64 - needs further investigation to [functional][systemd][u][fast] test fails in systemd_testsuite on aarch64 - needs further investigation
  • Due date set to 2018-06-19
  • Priority changed from Normal to Urgent
  • Target version set to Milestone 17

happening in SLE15 GMC.

#3 Updated by mgriessmeier about 5 years ago

  • Subject changed from [functional][systemd][u][fast] test fails in systemd_testsuite on aarch64 - needs further investigation to [functional][systemd][u][fast][medium] test fails in systemd_testsuite - needs further investigation
  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by SLindoMansilla about 5 years ago

  • Status changed from Workable to In Progress
  • Assignee set to SLindoMansilla

#5 Updated by SLindoMansilla almost 5 years ago

First SR to reorganize SUSE patches accepted: https://build.opensuse.org/request/show/615165

#6 Updated by SLindoMansilla almost 5 years ago

aarch64 issue is a bug and is handle in https://bugzilla.suse.com/show_bug.cgi?id=1097440

#7 Updated by SLindoMansilla almost 5 years ago

  • Description updated (diff)

SR to patch upstream bug: https://build.opensuse.org/request/show/616259
SR to patch openSUSE bug: https://build.opensuse.org/request/show/616468

Investigating s390x issue.

#8 Updated by SLindoMansilla almost 5 years ago

  • Status changed from In Progress to Blocked
  • Priority changed from Urgent to High

At the moment SLE15 is already on GMC phase. Adjusting prio.

I cannot investigate further due to a malfunction of the s390x-susekvm worker: https://progress.opensuse.org/issues/33202

#9 Updated by SLindoMansilla almost 5 years ago

  • Blocked by action #33202: [sle][functional][s390x][zkvm][u][hard] test fails in boot_to_desktop - still insufficient error reporting, black screen with mouse cursor - we all hate it (was: I hate it) added

#10 Updated by SLindoMansilla almost 5 years ago

  • Status changed from Blocked to In Progress

I forgot to change status after s390x worker was self-healed

#11 Updated by SLindoMansilla almost 5 years ago

Follow up for "ID_LIKE" patch to be fully back ported into SLE15: https://github.com/openSUSE/systemd/blob/SLE15/test/test-functions

#12 Updated by okurz almost 5 years ago

  • Target version changed from Milestone 17 to Milestone 17

#13 Updated by SLindoMansilla almost 5 years ago

  • Description updated (diff)
  • Priority changed from High to Normal

The problem is identified.

SLE15 is GM, so lowering priority.

#14 Updated by SLindoMansilla almost 5 years ago

  • Description updated (diff)
  • Assignee changed from SLindoMansilla to tsaupe

#15 Updated by SLindoMansilla almost 5 years ago

  • Blocked by action #25248: [tools][functional][systemd][u] openQA-workers for systemd department added

#16 Updated by mgriessmeier almost 5 years ago

  • Due date changed from 2018-06-19 to 2018-07-03

#17 Updated by okurz almost 5 years ago

  • Subject changed from [functional][systemd][u][fast][medium] test fails in systemd_testsuite - needs further investigation to [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation
  • Due date deleted (2018-07-03)
  • Target version changed from Milestone 17 to Milestone 20

So I discussed in detail with slindomansilla related to #34996 as well. AFAIU this ticket is for SLE in particular. Currently we focus on SLE12SP4 and for SLE15 the tests had been failing for a long time. This is why I delay the ticket plan to M20 as we should get openSUSE Tumbleweed in a stable state first and then revisit the situation for SLE tests. Whenever we see tests failing consistently and for a long time we should move the test scenario to the according "test development" job groups on the openQA server(s). Let's focus into #34996 . tsaupe of course the plan is only according to our planning as QSF (QA SLE functional) to look into this issue again later, milestone 20, which is 2018-11, but you are free to continue on this ticket as you like :)

#18 Updated by okurz almost 5 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

#19 Updated by okurz almost 5 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

#20 Updated by okurz almost 5 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

#21 Updated by okurz over 4 years ago

  • Status changed from In Progress to Workable
  • Assignee deleted (tsaupe)
  • Priority changed from Normal to High

Seems like no real progress here. By now I think we should take a look into SLE15 tests, at least remove the red blob from the validation tests. https://bugzilla.suse.com/show_bug.cgi?id=1105101 is currently used as label in these tests but also there nothing is moving forward, multiple unattended openqa-review reminder comments.

#22 Updated by nicksinger over 4 years ago

  • Related to action #32563: [functional][u] fix salt for power added

#23 Updated by SLindoMansilla over 4 years ago

  • Status changed from Workable to Feedback
  • Assignee set to SLindoMansilla

A SR provided by tsaupe was accepted yesterday: https://build.suse.de/request/show/175568

I will verify the fix on OSD.
Waiting for:

#24 Updated by SLindoMansilla over 4 years ago

  • Status changed from Feedback to In Progress

The job times out. I will try to reproduce how much time it needs to adapt the test.

#25 Updated by SLindoMansilla over 4 years ago

  • Status changed from In Progress to Feedback

#27 Updated by SLindoMansilla over 4 years ago

  • Status changed from Feedback to In Progress

Both fail. 1 hour timeout is not enough. Investigating...

#28 Updated by SLindoMansilla over 4 years ago

tblume has prepared a SR to solve this issue that makes the test hang in the qemu VM: https://build.suse.de/request/show/177535

#29 Updated by SLindoMansilla over 4 years ago

SR is accepted, waiting for the package to build on QA:SLE15: https://build.suse.de/package/show/QA:SLE15/systemd-v234-testsuite

#30 Updated by SLindoMansilla over 4 years ago

  • Status changed from In Progress to Feedback

Waiting for verification on OSD:

#31 Updated by SLindoMansilla over 4 years ago

It is still hanging on second test. Waiting for feedback from tblume.

#32 Updated by SLindoMansilla over 4 years ago

  • Has duplicate action #43826: [qam] [SLE15] test fails in systemd_testsuite added

#33 Updated by SLindoMansilla over 4 years ago

Last SR broke also x86_64 tests: https://openqa.suse.de/tests/2259623#step/systemd_testsuite/13

Reverting last SR

#34 Updated by SLindoMansilla over 4 years ago

tblume already provided a new SR with the fix: https://build.suse.de/request/show/177653

Not needed to revert changes.

#35 Updated by SLindoMansilla over 4 years ago

Waiting for verification on OSD:

#36 Updated by SLindoMansilla over 4 years ago

  • Status changed from Feedback to In Progress

Results for build 96.7:

Investigating fail on TEST-10:

#38 Updated by SLindoMansilla over 4 years ago

  • Status changed from In Progress to Feedback

New SR from tblume to fix aarch64 problem: https://build.suse.de/request/show/177815

Waiting for verification: https://openqa.suse.de/tests/2263888

#39 Updated by SLindoMansilla over 4 years ago

TEST-02 passes, but it fails on other tests later.

Verifying that the SR is not breaking s390x and x86_64 scenarios:

#40 Updated by okurz over 4 years ago

  • Related to action #44150: [functional][u] test fails in systemd_testsuite, Test died: Could not retrieve required variable QA_HEAD_REPO at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/console/systemd_testsuite.pm line 21. added

#41 Updated by okurz over 4 years ago

  • Priority changed from High to Urgent
  • Target version changed from Milestone 20 to Milestone 21

M20 is over, as we already shifted a lot I am not sure how this will go on. As we still have jobs failing labeled with this ticket, e.g. https://openqa.suse.de/tests/2271738#step/systemd_testsuite/17, I see it as urgent now. The urgency could be handled by removing the scenario again. But I leave it to you to decide which way to go, just … urgently please :)

#42 Updated by SLindoMansilla over 4 years ago

  • Status changed from Feedback to In Progress

At the moment there is no s390x worker to verify.
OSD ones are getting freeze.
QSF shared workers are not working because the s390x zkvm backend was never adapted to work on workers with enabled cache.

Fixing s390x backend first and moving aarch64 and ppc64le scenarios to development job group: https://openqa.suse.de/admin/job_templates/96

#43 Updated by SLindoMansilla over 4 years ago

  • Priority changed from Urgent to High

Urgency removed after moving aarch64 and ppc64le scenarios to development job group.

okurz, feel free to set normal prio.

#44 Updated by okurz over 4 years ago

  • Priority changed from High to Normal

yes, normal prio is fine.

#45 Updated by SLindoMansilla over 4 years ago

  • Status changed from In Progress to Blocked

#46 Updated by SLindoMansilla over 4 years ago

  • Status changed from Blocked to In Progress

#47 Updated by SLindoMansilla over 4 years ago

Generating the missing qcow2 image: http://slindomansilla-vm.qa.suse.de/tests/962

#48 Updated by SLindoMansilla over 4 years ago

  • Related to action #44468: [qe-core][functional][tools] Proper handling of assets for svirt workers added

#49 Updated by SLindoMansilla over 4 years ago

  • Related to deleted (action #44468: [qe-core][functional][tools] Proper handling of assets for svirt workers)

#50 Updated by SLindoMansilla over 4 years ago

  • Blocked by action #44468: [qe-core][functional][tools] Proper handling of assets for svirt workers added

#51 Updated by SLindoMansilla over 4 years ago

  • Blocked by deleted (action #25248: [tools][functional][systemd][u] openQA-workers for systemd department)

#52 Updated by SLindoMansilla over 4 years ago

  • Related to action #25248: [tools][functional][systemd][u] openQA-workers for systemd department added

#53 Updated by SLindoMansilla over 4 years ago

  • Status changed from In Progress to Feedback

s390x verification failed: https://openqa.suse.de/tests/2288353

Waiting for tsaupe's feedback on https://build.suse.de/request/show/177956

#54 Updated by SLindoMansilla over 4 years ago

  • Status changed from Feedback to Workable

Not able to verify on s390x until there is an available qcow2 image of SLE15-SP1 textmode for s390x-kvm-sle12

#55 Updated by SLindoMansilla over 4 years ago

  • Related to action #45158: [systemd] Implement systemd testsuite as openQA perl module added

#56 Updated by SLindoMansilla over 4 years ago

  • Status changed from Workable to Blocked

Blocked by #45158

#57 Updated by SLindoMansilla over 4 years ago

  • Related to deleted (action #45158: [systemd] Implement systemd testsuite as openQA perl module)

#58 Updated by SLindoMansilla over 4 years ago

  • Blocked by action #45158: [systemd] Implement systemd testsuite as openQA perl module added

#59 Updated by okurz over 4 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2343909

#60 Updated by okurz over 4 years ago

  • Target version changed from Milestone 21 to future

due to blocker

#61 Updated by okurz over 4 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2429581

#62 Updated by okurz over 4 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2452703

#63 Updated by SLindoMansilla over 4 years ago

The test suite was moved to the development job group due to the bug in existing tests that are taking long time to be fixed: https://openqa.suse.de/admin/job_templates/96

Latest: https://openqa.suse.de/tests/latest?version=15-SP1&distri=sle&test=suse_patches-systemd_testsuite&machine=64bit&arch=x86_64&flavor=Installer-DVD#next_previous

#65 Updated by SLindoMansilla about 4 years ago

  • Blocked by deleted (action #45158: [systemd] Implement systemd testsuite as openQA perl module)

#66 Updated by SLindoMansilla about 4 years ago

  • Related to action #45158: [systemd] Implement systemd testsuite as openQA perl module added

#67 Updated by tjyrinki_suse over 2 years ago

  • Subject changed from [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation to [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation

#68 Updated by SLindoMansilla about 2 years ago

  • Blocked by deleted (action #44468: [qe-core][functional][tools] Proper handling of assets for svirt workers)

#69 Updated by SLindoMansilla about 2 years ago

  • Related to action #44468: [qe-core][functional][tools] Proper handling of assets for svirt workers added

#70 Updated by SLindoMansilla about 2 years ago

  • Status changed from Blocked to Resolved

Issue not reproducible anymore

Also available in: Atom PDF