action #45158

[systemd] Implement systemd testsuite as openQA perl module

Added by SLindoMansilla about 1 year ago. Updated 8 months ago.

Status:In ProgressStart date:18/01/2019
Priority:NormalDue date:
Assignee:tsaupe% Done:

0%

Category:Enhancement to existing tests
Target version:QA - future
Difficulty:
Duration:

Description

Motivation

We are using the systemd's upstream test suite, but it is designed to run at build time. With some patches done by tsaupe, we are able to run the test suite in a running system.

Find the custom systemd-testsuite here:

In our effort to run automatic tests (openQA) on all 4 supported architectures (aarch64, ppc64le, s390x and x86_64), we have faced several issues.

suse_patches-systemd_testsuite:

Known problems

  • Nested virtualization (systemd-qemu's testsuite running on top of openQA qemu [even running on top of a VM s390x]) causes a lot of hangs on tests. In order to avoid this nested virtualization, we would need implement those test as an openQA perl module.
  • Test suite maintenance: The maintenance of the test suite used in openQA became very difficult.
    • Factory currently using v239: Starting from systemd-v237 (Tumbleweed), some test of the upstream test suite stopped working because they need a seed generated at build time for the systemd package. Since the custom systemd-testsuite by tsaupe is build in a separate OBS package, it doesn't have the same seed and the test fails to run. Find the OBS package here: https://build.opensuse.org/package/show/openSUSE:Factory/systemd
    • On development versions (Leap 15.1 and SLE 15-SP1 using systemd-v234) hangs on the test are reproducible as shown in previous jobs. Find the OBS package here: https://build.suse.de/package/show/home:fbui:systemd:SLE-15/systemd
    • On released versions, also maintenance (SLE 12-SP3, Leap 42.3) there is no OBS package, so this test suite is not executed on QAM for regressions qa_userspace_systemd is used instead)
    • See also related tickets, which could be resolved by this new approach.

Acceptance criteria

  • AC1: The upstream test suite is implemented as an openQA perl module to be executed as a normal test, not depending on any special OBS package.
  • AC2: Factory, development and maintenance openQA tests are able to use this new implemented module.

Related issues

Related to openQA Tests - action #42722: [sle][systemd] test fails in systemd_testsuite - Race con... New 19/10/2018
Related to openQA Tests - action #39041: [functional][sle][u][epic] test fails in systemd_testsuit... New 01/08/2018
Related to openQA Tests - action #35080: [functional][u] Enable the test suite for systemd on SLE1... New 17/04/2018
Related to openQA Tests - action #25210: [sle][functional][u] Add test suite patched_systemd New 12/09/2017 31/12/2019
Related to openQA Tests - action #36754: [functional][systemd][u][medium] test fails in systemd_te... Blocked 04/06/2018
Duplicated by openQA Tests - action #51170: test fails in binary_tests Rejected 06/05/2019
Blocks openQA Tests - action #23798: [sles][functional][u][epic] Add systemd test suite execut... Blocked 11/09/2017 31/12/2019
Blocks openQA Tests - action #34996: [functional][opensuse][u][epic] test fails in systemd_tes... Blocked 25/06/2018
Follows openQA Project - action #46349: [s390x] Add qemu backend support New 17/01/2019

History

#1 Updated by SLindoMansilla about 1 year ago

  • Related to action #42722: [sle][systemd] test fails in systemd_testsuite - Race condition of time sensitive tests on slow performance machines added

#2 Updated by SLindoMansilla about 1 year ago

  • Related to action #39041: [functional][sle][u][epic] test fails in systemd_testsuite - Adapt test suite 'suse_patches-systemd_testsuite' for SLE 12 added

#3 Updated by SLindoMansilla about 1 year ago

  • Related to action #35080: [functional][u] Enable the test suite for systemd on SLE12-SP4/5 added

#4 Updated by SLindoMansilla about 1 year ago

  • Related to action #25210: [sle][functional][u] Add test suite patched_systemd added

#5 Updated by SLindoMansilla about 1 year ago

  • Related to action #34996: [functional][opensuse][u][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added

#6 Updated by SLindoMansilla about 1 year ago

  • Related to action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation added

#7 Updated by SLindoMansilla about 1 year ago

  • Related to action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added

#8 Updated by mgriessmeier about 1 year ago

  • Blocks action #23798: [sles][functional][u][epic] Add systemd test suite execution to openQA added

#9 Updated by SLindoMansilla about 1 year ago

  • Related to deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)

#10 Updated by SLindoMansilla about 1 year ago

  • Blocks action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added

#11 Updated by SLindoMansilla about 1 year ago

  • Related to deleted (action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation)

#12 Updated by SLindoMansilla about 1 year ago

  • Blocks action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation added

#13 Updated by SLindoMansilla about 1 year ago

  • Related to deleted (action #34996: [functional][opensuse][u][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)

#14 Updated by SLindoMansilla about 1 year ago

  • Blocks action #34996: [functional][opensuse][u][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added

#15 Updated by SLindoMansilla about 1 year ago

New test suite added to O3: systemd-testsuite-01-basic
Test suite schedule in Tumbleweed Development job group: https://openqa.opensuse.org/group_overview/38

PR to have a template that tsaupe can use to start the migration: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6578

#16 Updated by SLindoMansilla about 1 year ago

#18 Updated by okurz 12 months ago

  • Subject changed from [functional][u] Implement systemd testsuite as openQA perl module to [systemd] Implement systemd testsuite as openQA perl module

#19 Updated by SLindoMansilla 10 months ago

  • Blocks deleted (action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation)

#20 Updated by SLindoMansilla 10 months ago

  • Related to action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation added

#21 Updated by SLindoMansilla 10 months ago

  • Status changed from New to In Progress

PR merged: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7247

Test suite setting updated in:

Waiting for verification on OSD and O3

#22 Updated by SLindoMansilla 10 months ago

#23 Updated by SLindoMansilla 10 months ago

#24 Updated by tsaupe 10 months ago

SLindoMansilla wrote:

Not working


Missing setting REPO_SLE_MODULE_DEVELOPMENT_TOOLS on:


I've updated my patch, to address that, please see:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7402

#27 Updated by tsaupe 8 months ago

SLindoMansilla wrote:

Preparing fix for the repository: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7688

Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.

#28 Updated by SLindoMansilla 8 months ago

tsaupe wrote:

Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.

I need to check the other architectures and will update my patch ASAP.

This was known. This was the reason of my crusade:

At the moment we a broken test for systemd, so enabling it for aarch64, ppc64le and x86_64 is at least something. We can take care of enabling it for s390x later.

The test suite is enabled only in test development job groups, so it is not critical if we push changes.

#29 Updated by tsaupe 8 months ago

ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276
Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?

#30 Updated by SLindoMansilla 8 months ago

tsaupe wrote:

ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276

awesome!

Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?

Your changes can be pushed. ;)

#33 Updated by mgriessmeier about 1 month ago

  • Blocks deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)

Also available in: Atom PDF