Project

General

Profile

Actions

action #45158

closed

[systemd] Implement systemd testsuite as openQA perl module

Added by SLindoMansilla about 5 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Enhancement to existing tests
Target version:
Start date:
2019-01-18
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

We are using the systemd's upstream test suite, but it is designed to run at build time. With some patches done by tsaupe, we are able to run the test suite in a running system.

Find the custom systemd-testsuite here:

In our effort to run automatic tests (openQA) on all 4 supported architectures (aarch64, ppc64le, s390x and x86_64), we have faced several issues.

suse_patches-systemd_testsuite:

Known problems

  • Nested virtualization (systemd-qemu's testsuite running on top of openQA qemu [even running on top of a VM s390x]) causes a lot of hangs on tests. In order to avoid this nested virtualization, we would need implement those test as an openQA perl module.
  • Test suite maintenance: The maintenance of the test suite used in openQA became very difficult.
    • Factory currently using v239: Starting from systemd-v237 (Tumbleweed), some test of the upstream test suite stopped working because they need a seed generated at build time for the systemd package. Since the custom systemd-testsuite by tsaupe is build in a separate OBS package, it doesn't have the same seed and the test fails to run. Find the OBS package here: https://build.opensuse.org/package/show/openSUSE:Factory/systemd
    • On development versions (Leap 15.1 and SLE 15-SP1 using systemd-v234) hangs on the test are reproducible as shown in previous jobs. Find the OBS package here: https://build.suse.de/package/show/home:fbui:systemd:SLE-15/systemd
    • On released versions, also maintenance (SLE 12-SP3, Leap 42.3) there is no OBS package, so this test suite is not executed on QAM for regressions qa_userspace_systemd is used instead)
    • See also related tickets, which could be resolved by this new approach.

Acceptance criteria

  • AC1: The upstream test suite is implemented as an openQA perl module to be executed as a normal test, not depending on any special OBS package.
  • AC2: Factory, development and maintenance openQA tests are able to use this new implemented module.

Related issues 8 (2 open6 closed)

Related to openQA Tests - action #42722: [sle][systemd] test fails in systemd_testsuite - Race condition of time sensitive tests on slow performance machinesNewtsaupe2018-10-19

Actions
Related to openQA Tests - coordination #39041: [functional][sle][u][epic] test fails in systemd_testsuite - Adapt test suite 'suse_patches-systemd_testsuite' for SLE 12RejectedSLindoMansilla2018-08-01

Actions
Related to openQA Tests - action #35080: [qe-core][functional][qem] Enable the test suite for systemd on SLE15-SP4Resolvedmloviska2018-04-17

Actions
Related to openQA Tests - action #25210: [sle][functional] Add test suite patched_systemdResolvedtsaupe2017-09-12

Actions
Related to openQA Tests - action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigationResolvedSLindoMansilla2018-06-04

Actions
Related to openQA Tests - coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQAResolvedSLindoMansilla2017-09-11

Actions
Has duplicate openQA Tests - action #51170: test fails in binary_testsRejectedSLindoMansilla2019-05-06

Actions
Follows openQA Project - action #46349: [s390x] Add qemu backend supportNew2019-01-17

Actions
Actions #1

Updated by SLindoMansilla about 5 years ago

  • Related to action #42722: [sle][systemd] test fails in systemd_testsuite - Race condition of time sensitive tests on slow performance machines added
Actions #2

Updated by SLindoMansilla about 5 years ago

  • Related to coordination #39041: [functional][sle][u][epic] test fails in systemd_testsuite - Adapt test suite 'suse_patches-systemd_testsuite' for SLE 12 added
Actions #3

Updated by SLindoMansilla about 5 years ago

  • Related to action #35080: [qe-core][functional][qem] Enable the test suite for systemd on SLE15-SP4 added
Actions #4

Updated by SLindoMansilla about 5 years ago

  • Related to action #25210: [sle][functional] Add test suite patched_systemd added
Actions #5

Updated by SLindoMansilla about 5 years ago

  • Related to coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added
Actions #6

Updated by SLindoMansilla about 5 years ago

  • Related to action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
Actions #7

Updated by SLindoMansilla about 5 years ago

  • Related to action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added
Actions #8

Updated by mgriessmeier about 5 years ago

  • Blocks coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA added
Actions #9

Updated by SLindoMansilla about 5 years ago

  • Related to deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)
Actions #10

Updated by SLindoMansilla about 5 years ago

  • Blocks action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added
Actions #11

Updated by SLindoMansilla about 5 years ago

  • Related to deleted (action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation)
Actions #12

Updated by SLindoMansilla about 5 years ago

  • Blocks action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
Actions #13

Updated by SLindoMansilla about 5 years ago

  • Related to deleted (coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)
Actions #14

Updated by SLindoMansilla about 5 years ago

  • Blocks coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added
Actions #15

Updated by SLindoMansilla about 5 years ago

New test suite added to O3: systemd-testsuite-01-basic
Test suite schedule in Tumbleweed Development job group: https://openqa.opensuse.org/group_overview/38

PR to have a template that tsaupe can use to start the migration: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6578

Actions #16

Updated by SLindoMansilla about 5 years ago

Actions #18

Updated by okurz almost 5 years ago

  • Subject changed from [functional][u] Implement systemd testsuite as openQA perl module to [systemd] Implement systemd testsuite as openQA perl module
Actions #19

Updated by SLindoMansilla almost 5 years ago

  • Blocks deleted (action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation)
Actions #20

Updated by SLindoMansilla almost 5 years ago

  • Related to action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
Actions #21

Updated by SLindoMansilla almost 5 years ago

  • Status changed from New to In Progress

PR merged: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7247

Test suite setting updated in:

Waiting for verification on OSD and O3

Actions #22

Updated by SLindoMansilla almost 5 years ago

Actions #24

Updated by tsaupe almost 5 years ago

SLindoMansilla wrote:

Not working

Missing setting REPO_SLE_MODULE_DEVELOPMENT_TOOLS on:

I've updated my patch, to address that, please see:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7402

Actions #27

Updated by tsaupe over 4 years ago

SLindoMansilla wrote:

Preparing fix for the repository: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7688

Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.

Actions #28

Updated by SLindoMansilla over 4 years ago

tsaupe wrote:

Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.

This was known. This was the reason of my crusade:

At the moment we a broken test for systemd, so enabling it for aarch64, ppc64le and x86_64 is at least something. We can take care of enabling it for s390x later.

The test suite is enabled only in test development job groups, so it is not critical if we push changes.

Actions #29

Updated by tsaupe over 4 years ago

ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276
Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?

Actions #30

Updated by SLindoMansilla over 4 years ago

tsaupe wrote:

ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276

awesome!

Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?

Your changes can be pushed. ;)

Actions #33

Updated by mgriessmeier about 4 years ago

  • Blocks deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)
Actions #34

Updated by mgriessmeier almost 4 years ago

  • Blocks deleted (coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)
Actions #35

Updated by SLindoMansilla almost 3 years ago

  • Blocks deleted (coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA)
Actions #36

Updated by SLindoMansilla almost 3 years ago

  • Related to coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA added
Actions #37

Updated by okurz over 2 years ago

  • Priority changed from Normal to Low

This ticket was set to "Normal" priority but was not updated within 730 days which is 2 times the period of the SLO for "Normal" tickets (365 days) as described on https://progress.opensuse.org/projects/openqatests/wiki/Wiki#SLOs-service-level-objectives . The ticket will be set to the next lower priority of "Low".

Actions #38

Updated by tsaupe over 2 years ago

The systemd testsuite has been removed from the official openqa tests.
We will run it on a team internal openQA machine instead.

Hence, I guess you can close this ticket.

Actions #39

Updated by szarate about 2 years ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF