Project

General

Profile

Actions
Copy link

action #36754

closed

[qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation

Added by nicksinger over 6 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
SLindoMansilla
Category:
Bugs in existing tests
Target version:
QA (public) - future
Start date:
2018-06-04
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

Could not access KVM kernel module: No such device

Reproducible

Acceptance criteria

AC1 Test suite suse_patches-systemd_testsuite doesn't fail on missing KVM kernel module

Tasks

  1. Enable KVM virtualization on openQA-worker hosts.
  2. If necessary, install SLE15 for aarch64 openQA-workers hosts (tblume is not sure if we can do it on SLE12)

Expected result

Last good build 563.1:

Related issues 8 (1 open — 7 closed)

Related to openQA Tests (public) - coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfileRejectedSLindoMansilla2018-06-25

Actions
Related to openQA Project (public) - action #32563: [functional][u] fix salt for powerResolvedszarate2018-02-28

Actions
Related to openQA Tests (public) - action #44150: [functional][u] test fails in systemd_testsuite, Test died: Could not retrieve required variable QA_HEAD_REPO at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/console/systemd_testsuite.pm line 21.ResolvedSLindoMansilla2018-11-21

Actions
Related to openQA Tests (public) - action #25248: [tools][functional][systemd][u] openQA-workers for systemd departmentResolvedtsaupe2017-09-13

Actions
Related to openQA Tests (public) - action #45158: [systemd] Implement systemd testsuite as openQA perl moduleResolvedtsaupe2019-01-18

Actions
Related to openQA Tests (public) - action #44468: [qe-core] Proper handling of assets for svirt workersWorkable2018-11-28

Actions
Has duplicate openQA Tests (public) - action #43826: [qam] [SLE15] test fails in systemd_testsuiteRejected2018-11-15

Actions
Blocked by openQA Tests (public) - action #33202: [sle][functional][s390x][zkvm][u][hard] test fails in boot_to_desktop - still insufficient error reporting, black screen with mouse cursor - we all hate it (was: I hate it)Resolvedmgriessmeier2018-03-132018-08-14

Actions
Actions
Copy link
 #1

Updated by SLindoMansilla over 6 years ago

  • Related to coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added
Actions
Copy link
 #2

Updated by okurz over 6 years ago

  • Subject changed from [functional][systemd][u] test fails in systemd_testsuite on aarch64 - needs further investigation to [functional][systemd][u][fast] test fails in systemd_testsuite on aarch64 - needs further investigation
  • Due date set to 2018-06-19
  • Priority changed from Normal to Urgent
  • Target version set to Milestone 17

happening in SLE15 GMC.

Actions
Copy link
 #3

Updated by mgriessmeier over 6 years ago

  • Subject changed from [functional][systemd][u][fast] test fails in systemd_testsuite on aarch64 - needs further investigation to [functional][systemd][u][fast][medium] test fails in systemd_testsuite - needs further investigation
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #4

Updated by SLindoMansilla over 6 years ago

  • Status changed from Workable to In Progress
  • Assignee set to SLindoMansilla
Actions
Copy link
 #5

Updated by SLindoMansilla over 6 years ago

First SR to reorganize SUSE patches accepted: https://build.opensuse.org/request/show/615165

Actions
Copy link
 #6

Updated by SLindoMansilla over 6 years ago

aarch64 issue is a bug and is handle in https://bugzilla.suse.com/show_bug.cgi?id=1097440

Actions
Copy link
 #7

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)

SR to patch upstream bug: https://build.opensuse.org/request/show/616259
SR to patch openSUSE bug: https://build.opensuse.org/request/show/616468

Investigating s390x issue.

Actions
Copy link
 #8

Updated by SLindoMansilla over 6 years ago

  • Status changed from In Progress to Blocked
  • Priority changed from Urgent to High

At the moment SLE15 is already on GMC phase. Adjusting prio.

I cannot investigate further due to a malfunction of the s390x-susekvm worker: https://progress.opensuse.org/issues/33202

Actions
Copy link
 #9

Updated by SLindoMansilla over 6 years ago

  • Blocked by action #33202: [sle][functional][s390x][zkvm][u][hard] test fails in boot_to_desktop - still insufficient error reporting, black screen with mouse cursor - we all hate it (was: I hate it) added
Actions
Copy link
 #10

Updated by SLindoMansilla over 6 years ago

  • Status changed from Blocked to In Progress

I forgot to change status after s390x worker was self-healed

Actions
Copy link
 #11

Updated by SLindoMansilla over 6 years ago

Follow up for "ID_LIKE" patch to be fully back ported into SLE15: https://github.com/openSUSE/systemd/blob/SLE15/test/test-functions

Actions
Copy link
 #12

Updated by okurz over 6 years ago

  • Target version changed from Milestone 17 to Milestone 17
Actions
Copy link
 #13

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
  • Priority changed from High to Normal

The problem is identified.

SLE15 is GM, so lowering priority.

Actions
Copy link
 #14

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
  • Assignee changed from SLindoMansilla to tsaupe
Actions
Copy link
 #15

Updated by SLindoMansilla over 6 years ago

  • Blocked by action #25248: [tools][functional][systemd][u] openQA-workers for systemd department added
Actions
Copy link
 #16

Updated by mgriessmeier over 6 years ago

  • Due date changed from 2018-06-19 to 2018-07-03
Actions
Copy link
 #17

Updated by okurz over 6 years ago

  • Subject changed from [functional][systemd][u][fast][medium] test fails in systemd_testsuite - needs further investigation to [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation
  • Due date deleted (2018-07-03)
  • Target version changed from Milestone 17 to Milestone 20

So I discussed in detail with slindomansilla related to #34996 as well. AFAIU this ticket is for SLE in particular. Currently we focus on SLE12SP4 and for SLE15 the tests had been failing for a long time. This is why I delay the ticket plan to M20 as we should get openSUSE Tumbleweed in a stable state first and then revisit the situation for SLE tests. Whenever we see tests failing consistently and for a long time we should move the test scenario to the according "test development" job groups on the openQA server(s). Let's focus into #34996 . @tsaupe of course the plan is only according to our planning as QSF (QA SLE functional) to look into this issue again later, milestone 20, which is 2018-11, but you are free to continue on this ticket as you like :)

Actions
Copy link
 #18

Updated by okurz over 6 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

Actions
Copy link
 #19

Updated by okurz over 6 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

Actions
Copy link
 #20

Updated by okurz over 6 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/1772093

Actions
Copy link
 #21

Updated by okurz about 6 years ago

  • Status changed from In Progress to Workable
  • Assignee deleted (tsaupe)
  • Priority changed from Normal to High

Seems like no real progress here. By now I think we should take a look into SLE15 tests, at least remove the red blob from the validation tests. https://bugzilla.suse.com/show_bug.cgi?id=1105101 is currently used as label in these tests but also there nothing is moving forward, multiple unattended openqa-review reminder comments.

Actions
Copy link
 #22

Updated by nicksinger about 6 years ago

  • Related to action #32563: [functional][u] fix salt for power added
Actions
Copy link
 #23

Updated by SLindoMansilla about 6 years ago

  • Status changed from Workable to Feedback
  • Assignee set to SLindoMansilla

A SR provided by tsaupe was accepted yesterday: https://build.suse.de/request/show/175568

I will verify the fix on OSD.
Waiting for:

Actions
Copy link
 #24

Updated by SLindoMansilla about 6 years ago

  • Status changed from Feedback to In Progress

The job times out. I will try to reproduce how much time it needs to adapt the test.

Actions
Copy link
 #25

Updated by SLindoMansilla about 6 years ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #27

Updated by SLindoMansilla about 6 years ago

  • Status changed from Feedback to In Progress

Both fail. 1 hour timeout is not enough. Investigating...

Actions
Copy link
 #28

Updated by SLindoMansilla about 6 years ago

tblume has prepared a SR to solve this issue that makes the test hang in the qemu VM: https://build.suse.de/request/show/177535

Actions
Copy link
 #29

Updated by SLindoMansilla about 6 years ago

SR is accepted, waiting for the package to build on QA:SLE15: https://build.suse.de/package/show/QA:SLE15/systemd-v234-testsuite

Actions
Copy link
 #30

Updated by SLindoMansilla about 6 years ago

  • Status changed from In Progress to Feedback

Waiting for verification on OSD:

Actions
Copy link
 #31

Updated by SLindoMansilla about 6 years ago

It is still hanging on second test. Waiting for feedback from tblume.

Actions
Copy link
 #32

Updated by SLindoMansilla about 6 years ago

  • Has duplicate action #43826: [qam] [SLE15] test fails in systemd_testsuite added
Actions
Copy link
 #33

Updated by SLindoMansilla about 6 years ago

Last SR broke also x86_64 tests: https://openqa.suse.de/tests/2259623#step/systemd_testsuite/13

Reverting last SR

Actions
Copy link
 #34

Updated by SLindoMansilla about 6 years ago

tblume already provided a new SR with the fix: https://build.suse.de/request/show/177653

Not needed to revert changes.

Actions
Copy link
 #35

Updated by SLindoMansilla about 6 years ago

Waiting for verification on OSD:

Actions
Copy link
 #36

Updated by SLindoMansilla about 6 years ago

  • Status changed from Feedback to In Progress

Results for build 96.7:

Investigating fail on TEST-10:

Actions
Copy link
 #38

Updated by SLindoMansilla about 6 years ago

  • Status changed from In Progress to Feedback

New SR from tblume to fix aarch64 problem: https://build.suse.de/request/show/177815

Waiting for verification: https://openqa.suse.de/tests/2263888

Actions
Copy link
 #39

Updated by SLindoMansilla about 6 years ago

TEST-02 passes, but it fails on other tests later.

Verifying that the SR is not breaking s390x and x86_64 scenarios:

Actions
Copy link
 #40

Updated by okurz about 6 years ago

  • Related to action #44150: [functional][u] test fails in systemd_testsuite, Test died: Could not retrieve required variable QA_HEAD_REPO at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/console/systemd_testsuite.pm line 21. added
Actions
Copy link
 #41

Updated by okurz about 6 years ago

  • Priority changed from High to Urgent
  • Target version changed from Milestone 20 to Milestone 21

M20 is over, as we already shifted a lot I am not sure how this will go on. As we still have jobs failing labeled with this ticket, e.g. https://openqa.suse.de/tests/2271738#step/systemd_testsuite/17, I see it as urgent now. The urgency could be handled by removing the scenario again. But I leave it to you to decide which way to go, just … urgently please :)

Actions
Copy link
 #42

Updated by SLindoMansilla about 6 years ago

  • Status changed from Feedback to In Progress

At the moment there is no s390x worker to verify.
OSD ones are getting freeze.
QSF shared workers are not working because the s390x zkvm backend was never adapted to work on workers with enabled cache.

Fixing s390x backend first and moving aarch64 and ppc64le scenarios to development job group: https://openqa.suse.de/admin/job_templates/96

Actions
Copy link
 #43

Updated by SLindoMansilla about 6 years ago

  • Priority changed from Urgent to High

Urgency removed after moving aarch64 and ppc64le scenarios to development job group.

@okurz, feel free to set normal prio.

Actions
Copy link
 #44

Updated by okurz about 6 years ago

  • Priority changed from High to Normal

yes, normal prio is fine.

Actions
Copy link
 #45

Updated by SLindoMansilla about 6 years ago

  • Status changed from In Progress to Blocked
Actions
Copy link
 #46

Updated by SLindoMansilla about 6 years ago

  • Status changed from Blocked to In Progress
Actions
Copy link
 #47

Updated by SLindoMansilla about 6 years ago

Generating the missing qcow2 image: http://slindomansilla-vm.qa.suse.de/tests/962

Actions
Copy link
 #48

Updated by SLindoMansilla about 6 years ago

  • Related to action #44468: [qe-core] Proper handling of assets for svirt workers added
Actions
Copy link
 #49

Updated by SLindoMansilla about 6 years ago

  • Related to deleted (action #44468: [qe-core] Proper handling of assets for svirt workers)
Actions
Copy link
 #50

Updated by SLindoMansilla about 6 years ago

  • Blocked by action #44468: [qe-core] Proper handling of assets for svirt workers added
Actions
Copy link
 #51

Updated by SLindoMansilla about 6 years ago

  • Blocked by deleted (action #25248: [tools][functional][systemd][u] openQA-workers for systemd department)
Actions
Copy link
 #52

Updated by SLindoMansilla about 6 years ago

  • Related to action #25248: [tools][functional][systemd][u] openQA-workers for systemd department added
Actions
Copy link
 #53

Updated by SLindoMansilla about 6 years ago

  • Status changed from In Progress to Feedback

s390x verification failed: https://openqa.suse.de/tests/2288353

Waiting for tsaupe's feedback on https://build.suse.de/request/show/177956

Actions
Copy link
 #54

Updated by SLindoMansilla about 6 years ago

  • Status changed from Feedback to Workable

Not able to verify on s390x until there is an available qcow2 image of SLE15-SP1 textmode for s390x-kvm-sle12

Actions
Copy link
 #55

Updated by SLindoMansilla about 6 years ago

  • Related to action #45158: [systemd] Implement systemd testsuite as openQA perl module added
Actions
Copy link
 #56

Updated by SLindoMansilla about 6 years ago

  • Status changed from Workable to Blocked

Blocked by #45158

Actions
Copy link
 #57

Updated by SLindoMansilla about 6 years ago

  • Related to deleted (action #45158: [systemd] Implement systemd testsuite as openQA perl module)
Actions
Copy link
 #58

Updated by SLindoMansilla about 6 years ago

  • Blocked by action #45158: [systemd] Implement systemd testsuite as openQA perl module added
Actions
Copy link
 #59

Updated by okurz almost 6 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2343909

Actions
Copy link
 #60

Updated by okurz almost 6 years ago

  • Target version changed from Milestone 21 to future

due to blocker

Actions
Copy link
 #61

Updated by okurz almost 6 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2429581

Actions
Copy link
 #62

Updated by okurz almost 6 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: suse_patches-systemd_testsuite
https://openqa.suse.de/tests/2452703

Actions
Copy link
 #63

Updated by SLindoMansilla almost 6 years ago

The test suite was moved to the development job group due to the bug in existing tests that are taking long time to be fixed: https://openqa.suse.de/admin/job_templates/96

Latest: https://openqa.suse.de/tests/latest?version=15-SP1&distri=sle&test=suse_patches-systemd_testsuite&machine=64bit&arch=x86_64&flavor=Installer-DVD#next_previous

Actions
Copy link
 #65

Updated by SLindoMansilla over 5 years ago

  • Blocked by deleted (action #45158: [systemd] Implement systemd testsuite as openQA perl module)
Actions
Copy link
 #66

Updated by SLindoMansilla over 5 years ago

  • Related to action #45158: [systemd] Implement systemd testsuite as openQA perl module added
Actions
Copy link
 #67

Updated by tjyrinki_suse about 4 years ago

  • Subject changed from [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation to [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation
Actions
Copy link
 #68

Updated by SLindoMansilla almost 4 years ago

  • Blocked by deleted (action #44468: [qe-core] Proper handling of assets for svirt workers)
Actions
Copy link
 #69

Updated by SLindoMansilla almost 4 years ago

  • Related to action #44468: [qe-core] Proper handling of assets for svirt workers added
Actions
Copy link
 #70

Updated by SLindoMansilla almost 4 years ago

  • Status changed from Blocked to Resolved

Issue not reproducible anymore

Actions
Copy link

Also available in: Atom PDF