action #45158
closed[systemd] Implement systemd testsuite as openQA perl module
0%
Description
Motivation¶
We are using the systemd's upstream test suite, but it is designed to run at build time. With some patches done by tsaupe, we are able to run the test suite in a running system.
Find the custom systemd-testsuite here:
- https://build.opensuse.org/package/show/devel:openSUSE:QA:Tumbleweed/systemd-v237-testsuite
- https://build.suse.de/package/show/QA:SLE15/systemd-v234-testsuite
In our effort to run automatic tests (openQA) on all 4 supported architectures (aarch64, ppc64le, s390x and x86_64), we have faced several issues.
suse_patches-systemd_testsuite:¶
- Tumbleweed#latest
- SLE15-SP1-aarch64#latest
- SLE15-SP1-ppc64le#latest
- SLE15-SP1-s390x#latest
- SLE15-SP1-x86_64#latest
Known problems¶
- Nested virtualization (systemd-qemu's testsuite running on top of openQA qemu [even running on top of a VM s390x]) causes a lot of hangs on tests. In order to avoid this nested virtualization, we would need implement those test as an openQA perl module.
- Test suite maintenance: The maintenance of the test suite used in openQA became very difficult.
- Factory currently using v239: Starting from systemd-v237 (Tumbleweed), some test of the upstream test suite stopped working because they need a seed generated at build time for the systemd package. Since the custom systemd-testsuite by tsaupe is build in a separate OBS package, it doesn't have the same seed and the test fails to run. Find the OBS package here: https://build.opensuse.org/package/show/openSUSE:Factory/systemd
- On development versions (Leap 15.1 and SLE 15-SP1 using systemd-v234) hangs on the test are reproducible as shown in previous jobs. Find the OBS package here: https://build.suse.de/package/show/home:fbui:systemd:SLE-15/systemd
- On released versions, also maintenance (SLE 12-SP3, Leap 42.3) there is no OBS package, so this test suite is not executed on QAM for regressions qa_userspace_systemd is used instead)
- See also related tickets, which could be resolved by this new approach.
Acceptance criteria¶
- AC1: The upstream test suite is implemented as an openQA perl module to be executed as a normal test, not depending on any special OBS package.
- AC2: Factory, development and maintenance openQA tests are able to use this new implemented module.
Updated by SLindoMansilla over 5 years ago
- Related to action #42722: [sle][systemd] test fails in systemd_testsuite - Race condition of time sensitive tests on slow performance machines added
Updated by SLindoMansilla over 5 years ago
- Related to coordination #39041: [functional][sle][u][epic] test fails in systemd_testsuite - Adapt test suite 'suse_patches-systemd_testsuite' for SLE 12 added
Updated by SLindoMansilla over 5 years ago
- Related to action #35080: [qe-core][functional][qem] Enable the test suite for systemd on SLE15-SP4 added
Updated by SLindoMansilla over 5 years ago
- Related to action #25210: [sle][functional] Add test suite patched_systemd added
Updated by SLindoMansilla over 5 years ago
- Related to coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added
Updated by SLindoMansilla over 5 years ago
- Related to action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
Updated by SLindoMansilla over 5 years ago
- Related to action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added
Updated by mgriessmeier over 5 years ago
- Blocks coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA added
Updated by SLindoMansilla over 5 years ago
- Related to deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)
Updated by SLindoMansilla over 5 years ago
- Blocks action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added
Updated by SLindoMansilla over 5 years ago
- Related to deleted (action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation)
Updated by SLindoMansilla over 5 years ago
- Blocks action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
Updated by SLindoMansilla over 5 years ago
- Related to deleted (coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)
Updated by SLindoMansilla over 5 years ago
- Blocks coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added
Updated by SLindoMansilla over 5 years ago
New test suite added to O3: systemd-testsuite-01-basic
Test suite schedule in Tumbleweed Development job group: https://openqa.opensuse.org/group_overview/38
PR to have a template that tsaupe can use to start the migration: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6578
Updated by SLindoMansilla over 5 years ago
- Follows action #46349: [s390x] Add qemu backend support added
Updated by SLindoMansilla over 5 years ago
- Assignee set to tsaupe
PR for migration: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6593
Updated by okurz about 5 years ago
- Subject changed from [functional][u] Implement systemd testsuite as openQA perl module to [systemd] Implement systemd testsuite as openQA perl module
Updated by SLindoMansilla almost 5 years ago
- Blocks deleted (action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation)
Updated by SLindoMansilla almost 5 years ago
- Related to action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
Updated by SLindoMansilla almost 5 years ago
- Status changed from New to In Progress
PR merged: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7247
Test suite setting updated in:
- OSD https://openqa.suse.de/admin/test_suites (suse_patches-systemd_testsuite)
- Scheduled in Test Development: SLE 15
- O3 https://openqa.opensuse.org/admin/test_suites (suse_patches-systemd_testsuite)
- Scheduled in Development Tumbleweed
- Scheduled in Development Leap
Waiting for verification on OSD and O3
Updated by SLindoMansilla almost 5 years ago
- Has duplicate action #51170: test fails in binary_tests added
Updated by SLindoMansilla almost 5 years ago
Not working
Missing setting REPO_SLE_MODULE_DEVELOPMENT_TOOLS on:
Updated by tsaupe almost 5 years ago
SLindoMansilla wrote:
Not working
Missing setting REPO_SLE_MODULE_DEVELOPMENT_TOOLS on:
I've updated my patch, to address that, please see:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7402
Updated by SLindoMansilla almost 5 years ago
Updated by SLindoMansilla almost 5 years ago
Preparing fix for the repository: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7688
Updated by tsaupe almost 5 years ago
SLindoMansilla wrote:
Preparing fix for the repository: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7688
Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.
Updated by SLindoMansilla almost 5 years ago
tsaupe wrote:
Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.
This was known. This was the reason of my crusade:
At the moment we a broken test for systemd, so enabling it for aarch64, ppc64le and x86_64 is at least something. We can take care of enabling it for s390x later.
The test suite is enabled only in test development job groups, so it is not critical if we push changes.
Updated by tsaupe almost 5 years ago
ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276
Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?
Updated by SLindoMansilla almost 5 years ago
tsaupe wrote:
ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276
awesome!
Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?
Your changes can be pushed. ;)
Updated by SLindoMansilla almost 5 years ago
New improvements: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7863
Updated by SLindoMansilla almost 5 years ago
More improvements: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7864
Updated by mgriessmeier over 4 years ago
- Blocks deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)
Updated by mgriessmeier about 4 years ago
- Blocks deleted (coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)
Updated by SLindoMansilla about 3 years ago
- Blocks deleted (coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA)
Updated by SLindoMansilla about 3 years ago
- Related to coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA added
Updated by okurz over 2 years ago
- Priority changed from Normal to Low
This ticket was set to "Normal" priority but was not updated within 730 days which is 2 times the period of the SLO for "Normal" tickets (365 days) as described on https://progress.opensuse.org/projects/openqatests/wiki/Wiki#SLOs-service-level-objectives . The ticket will be set to the next lower priority of "Low".
Updated by tsaupe over 2 years ago
The systemd testsuite has been removed from the official openqa tests.
We will run it on a team internal openQA machine instead.
Hence, I guess you can close this ticket.
Updated by szarate about 2 years ago
- Status changed from In Progress to Resolved