Project

General

Profile

action #45158

[systemd] Implement systemd testsuite as openQA perl module

Added by SLindoMansilla almost 2 years ago. Updated about 1 year ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Target version:
Start date:
2019-01-18
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

We are using the systemd's upstream test suite, but it is designed to run at build time. With some patches done by tsaupe, we are able to run the test suite in a running system.

Find the custom systemd-testsuite here:

In our effort to run automatic tests (openQA) on all 4 supported architectures (aarch64, ppc64le, s390x and x86_64), we have faced several issues.

suse_patches-systemd_testsuite:

Known problems

  • Nested virtualization (systemd-qemu's testsuite running on top of openQA qemu [even running on top of a VM s390x]) causes a lot of hangs on tests. In order to avoid this nested virtualization, we would need implement those test as an openQA perl module.
  • Test suite maintenance: The maintenance of the test suite used in openQA became very difficult.
    • Factory currently using v239: Starting from systemd-v237 (Tumbleweed), some test of the upstream test suite stopped working because they need a seed generated at build time for the systemd package. Since the custom systemd-testsuite by tsaupe is build in a separate OBS package, it doesn't have the same seed and the test fails to run. Find the OBS package here: https://build.opensuse.org/package/show/openSUSE:Factory/systemd
    • On development versions (Leap 15.1 and SLE 15-SP1 using systemd-v234) hangs on the test are reproducible as shown in previous jobs. Find the OBS package here: https://build.suse.de/package/show/home:fbui:systemd:SLE-15/systemd
    • On released versions, also maintenance (SLE 12-SP3, Leap 42.3) there is no OBS package, so this test suite is not executed on QAM for regressions qa_userspace_systemd is used instead)
    • See also related tickets, which could be resolved by this new approach.

Acceptance criteria

  • AC1: The upstream test suite is implemented as an openQA perl module to be executed as a normal test, not depending on any special OBS package.
  • AC2: Factory, development and maintenance openQA tests are able to use this new implemented module.

Related issues

Related to openQA Tests - action #42722: [sle][systemd] test fails in systemd_testsuite - Race condition of time sensitive tests on slow performance machinesNew2018-10-19

Related to openQA Tests - action #39041: [functional][sle][u][epic] test fails in systemd_testsuite - Adapt test suite 'suse_patches-systemd_testsuite' for SLE 12Rejected2018-08-01

Related to openQA Tests - action #35080: [functional][qam] Enable the test suite for systemd on SLE12-SP4/5New2018-04-17

Related to openQA Tests - action #25210: [sle][functional] Add test suite patched_systemdResolved2017-09-12

Related to openQA Tests - action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigationBlocked2018-06-04

Has duplicate openQA Tests - action #51170: test fails in binary_testsRejected2019-05-06

Blocks openQA Tests - action #23798: [sles][functional][u][epic] Add systemd test suite execution to openQABlocked2017-09-11

Follows openQA Project - action #46349: [s390x] Add qemu backend supportNew2019-01-17

History

#1 Updated by SLindoMansilla almost 2 years ago

  • Related to action #42722: [sle][systemd] test fails in systemd_testsuite - Race condition of time sensitive tests on slow performance machines added

#2 Updated by SLindoMansilla almost 2 years ago

  • Related to action #39041: [functional][sle][u][epic] test fails in systemd_testsuite - Adapt test suite 'suse_patches-systemd_testsuite' for SLE 12 added

#3 Updated by SLindoMansilla almost 2 years ago

  • Related to action #35080: [functional][qam] Enable the test suite for systemd on SLE12-SP4/5 added

#4 Updated by SLindoMansilla almost 2 years ago

  • Related to action #25210: [sle][functional] Add test suite patched_systemd added

#5 Updated by SLindoMansilla almost 2 years ago

  • Related to action #34996: [functional][opensuse][u][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added

#6 Updated by SLindoMansilla almost 2 years ago

  • Related to action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation added

#7 Updated by SLindoMansilla almost 2 years ago

  • Related to action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added

#8 Updated by mgriessmeier almost 2 years ago

  • Blocks action #23798: [sles][functional][u][epic] Add systemd test suite execution to openQA added

#9 Updated by SLindoMansilla almost 2 years ago

  • Related to deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)

#10 Updated by SLindoMansilla almost 2 years ago

  • Blocks action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added

#11 Updated by SLindoMansilla almost 2 years ago

  • Related to deleted (action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation)

#12 Updated by SLindoMansilla almost 2 years ago

  • Blocks action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation added

#13 Updated by SLindoMansilla almost 2 years ago

  • Related to deleted (action #34996: [functional][opensuse][u][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)

#14 Updated by SLindoMansilla almost 2 years ago

  • Blocks action #34996: [functional][opensuse][u][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added

#15 Updated by SLindoMansilla over 1 year ago

New test suite added to O3: systemd-testsuite-01-basic
Test suite schedule in Tumbleweed Development job group: https://openqa.opensuse.org/group_overview/38

PR to have a template that tsaupe can use to start the migration: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6578

#16 Updated by SLindoMansilla over 1 year ago

#18 Updated by okurz over 1 year ago

  • Subject changed from [functional][u] Implement systemd testsuite as openQA perl module to [systemd] Implement systemd testsuite as openQA perl module

#19 Updated by SLindoMansilla over 1 year ago

  • Blocks deleted (action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation)

#20 Updated by SLindoMansilla over 1 year ago

  • Related to action #36754: [functional][systemd][u][medium] test fails in systemd_testsuite - needs further investigation added

#21 Updated by SLindoMansilla over 1 year ago

  • Status changed from New to In Progress

PR merged: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7247

Test suite setting updated in:

Waiting for verification on OSD and O3

#22 Updated by SLindoMansilla over 1 year ago

#24 Updated by tsaupe over 1 year ago

SLindoMansilla wrote:

Not working

Missing setting REPO_SLE_MODULE_DEVELOPMENT_TOOLS on:

I've updated my patch, to address that, please see:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7402

#27 Updated by tsaupe over 1 year ago

SLindoMansilla wrote:

Preparing fix for the repository: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7688

Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.

#28 Updated by SLindoMansilla over 1 year ago

tsaupe wrote:

Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.

This was known. This was the reason of my crusade:

At the moment we a broken test for systemd, so enabling it for aarch64, ppc64le and x86_64 is at least something. We can take care of enabling it for s390x later.

The test suite is enabled only in test development job groups, so it is not critical if we push changes.

#29 Updated by tsaupe over 1 year ago

ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276
Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?

#30 Updated by SLindoMansilla over 1 year ago

tsaupe wrote:

ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276

awesome!

Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?

Your changes can be pushed. ;)

#33 Updated by mgriessmeier 8 months ago

  • Blocks deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)

#34 Updated by mgriessmeier 7 months ago

  • Blocks deleted (action #34996: [functional][opensuse][u][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)

Also available in: Atom PDF