action #45158
closed
[systemd] Implement systemd testsuite as openQA perl module
Added by SLindoMansilla about 6 years ago.
Updated almost 3 years ago.
Category:
Enhancement to existing tests
Description
Motivation¶
We are using the systemd's upstream test suite, but it is designed to run at build time. With some patches done by tsaupe, we are able to run the test suite in a running system.
Find the custom systemd-testsuite here:
In our effort to run automatic tests (openQA) on all 4 supported architectures (aarch64, ppc64le, s390x and x86_64), we have faced several issues.
suse_patches-systemd_testsuite:¶
Known problems¶
- Nested virtualization (systemd-qemu's testsuite running on top of openQA qemu [even running on top of a VM s390x]) causes a lot of hangs on tests. In order to avoid this nested virtualization, we would need implement those test as an openQA perl module.
- Test suite maintenance: The maintenance of the test suite used in openQA became very difficult.
- Factory currently using v239: Starting from systemd-v237 (Tumbleweed), some test of the upstream test suite stopped working because they need a seed generated at build time for the systemd package. Since the custom systemd-testsuite by tsaupe is build in a separate OBS package, it doesn't have the same seed and the test fails to run. Find the OBS package here: https://build.opensuse.org/package/show/openSUSE:Factory/systemd
- On development versions (Leap 15.1 and SLE 15-SP1 using systemd-v234) hangs on the test are reproducible as shown in previous jobs. Find the OBS package here: https://build.suse.de/package/show/home:fbui:systemd:SLE-15/systemd
- On released versions, also maintenance (SLE 12-SP3, Leap 42.3) there is no OBS package, so this test suite is not executed on QAM for regressions qa_userspace_systemd is used instead)
- See also related tickets, which could be resolved by this new approach.
Acceptance criteria¶
- AC1: The upstream test suite is implemented as an openQA perl module to be executed as a normal test, not depending on any special OBS package.
- AC2: Factory, development and maintenance openQA tests are able to use this new implemented module.
- Related to action #42722: [sle][systemd] test fails in systemd_testsuite - Race condition of time sensitive tests on slow performance machines added
- Related to coordination #39041: [functional][sle][u][epic] test fails in systemd_testsuite - Adapt test suite 'suse_patches-systemd_testsuite' for SLE 12 added
- Related to action #35080: [qe-core][functional][qem] Enable the test suite for systemd on SLE15-SP4 added
- Related to action #25210: [sle][functional] Add test suite patched_systemd added
- Related to coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added
- Related to action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
- Related to action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added
- Blocks coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA added
- Related to deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)
- Blocks action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1 added
- Related to deleted (action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation)
- Blocks action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
- Related to deleted (coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)
- Blocks coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile added
- Subject changed from [functional][u] Implement systemd testsuite as openQA perl module to [systemd] Implement systemd testsuite as openQA perl module
- Blocks deleted (action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation)
- Related to action #36754: [qe-core][functional][systemd][medium] test fails in systemd_testsuite - needs further investigation added
- Status changed from New to In Progress
Not working
Missing setting REPO_SLE_MODULE_DEVELOPMENT_TOOLS on:
SLindoMansilla wrote:
Preparing fix for the repository: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7688
Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.
tsaupe wrote:
Please hold it, I've found that vm snapshots are not supported on s390x, so our rollback approach after each test doesn't work, at least on s390.
I need to check the other architectures and will update my patch ASAP.
This was known. This was the reason of my crusade:
At the moment we a broken test for systemd, so enabling it for aarch64, ppc64le and x86_64 is at least something. We can take care of enabling it for s390x later.
The test suite is enabled only in test development job groups, so it is not critical if we push changes.
ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276
Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?
tsaupe wrote:
ok, meanwhile I've corrected most of the s390x tests, see: http://emerson.suse.de/tests/1276
awesome!
Do you want me to push my latest changes before you run the test or should I wait until you've tested with the current patches?
Your changes can be pushed. ;)
- Blocks deleted (action #40337: [functional][u] test failing to add systemd test repo - missing QA repo for Leap 15.1)
- Blocks deleted (coordination #34996: [qe-core][functional][opensuse][epic] test fails in systemd_testsuite - TEST-16-EXTEND-TIMEOUT works only when executed against systemd built in the same specfile)
- Blocks deleted (coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA)
- Related to coordination #23798: [qe-core][sles][functional][epic] Add systemd test suite execution to openQA added
- Priority changed from Normal to Low
The systemd testsuite has been removed from the official openqa tests.
We will run it on a team internal openQA machine instead.
Hence, I guess you can close this ticket.
- Status changed from In Progress to Resolved
Also available in: Atom
PDF