Project

General

Profile

Actions

action #151738

closed

[qe-core] Enable systemd-testsuite in 15-SP6

Added by jlausuch about 1 year ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
New test
Target version:
QA (public, currently private due to #173521) - QE-Core: Ready
Start date:
2023-11-30
Due date:
% Done:

100%

Estimated time:
Difficulty:
Tags:

Description

Background

There is an important update for systemd in 15-SP6 to v254: https://jira.suse.com/browse/PED-4846
Some important partners are a bit worried about this update and have asked us to please make sure this doesn't break any application (e.g. SAP applications).
We can't ensure 3rd party applications will be ok with this update, but what we can do is to provide good coverage for systemd testing (of course, SAP part will be covered by SAP squad).

systemd-testsuite has been integrated in openQA and enabled in TW by Martin Loviska when he was on rotation. e.g. https://openqa.opensuse.org/tests/3760221
It was also enabled in 15-SP4 and SP5 after GA, but the package is already in 15-SP6 channel http://download.suse.de/download/ibs/SUSE:/SLE-15-SP6:/GA/standard/x86_64/systemd-testsuite-254.5-150600.1.2.x86_64.rpm

Acceptance criteria

  • systemd-testsuite is scheduled in 15-SP6 functional group.

Related slack thead: https://suse.slack.com/archives/C02CSAZLAR4/p1701087663691749

Actions #1

Updated by szarate 12 months ago

  • Sprint set to QE-Core: December Sprint 23 (Dec 13 - Jan 10)
  • Tags changed from systemd to systemd, qe-core-december-sprint
  • Status changed from New to Workable
  • Target version set to QE-Core: Ready
Actions #2

Updated by szarate 12 months ago

  • Sprint deleted (QE-Core: December Sprint 23 (Dec 13 - Jan 10))
  • Tags changed from systemd, qe-core-december-sprint to systemd
Actions #3

Updated by amanzini 12 months ago

  • Status changed from Workable to In Progress
  • Assignee set to amanzini
Actions #4

Updated by amanzini 12 months ago · Edited

which architectures are being covered ? I got a missing dependency error installing the package in aarch64

Problem: nothing provides 'qemu-kvm' needed by the to be installed systemd-testsuite-254.5-150600.1.9.aarch64
 Solution 1: do not install systemd-testsuite-254.5-150600.1.9.aarch64
 Solution 2: break systemd-testsuite-254.5-150600.1.9.aarch64 by ignoring some of its dependencies

while on ppc64le

Activating sle-module-python3 15.6 ppc64le ...
Error: Registration server returned 'The product you are attempting to activate (Python 3 Module 15 SP6 ppc64le) is not available on your system's base product (SUSE Linux Enterprise Server 15 SP6 ppc64le). Python 3 Module 15 SP6 ppc64le is available on SUSE Linux Enterprise Server 15 SP5 ppc64le, SUSE Linux Enterprise Server for SAP Applications 15 SP5 ppc64le.' (422)
eTLB5-67-
Actions #5

Updated by rfan1 12 months ago

amanzini wrote in #note-4:

which architectures are being covered ? I got a missing dependency error installing the package in aarch64

Problem: nothing provides 'qemu-kvm' needed by the to be installed systemd-testsuite-254.5-150600.1.9.aarch64
 Solution 1: do not install systemd-testsuite-254.5-150600.1.9.aarch64
 Solution 2: break systemd-testsuite-254.5-150600.1.9.aarch64 by ignoring some of its dependencies

while on ppc64le

Activating sle-module-python3 15.6 ppc64le ...
Error: Registration server returned 'The product you are attempting to activate (Python 3 Module 15 SP6 ppc64le) is not available on your system's base product (SUSE Linux Enterprise Server 15 SP6 ppc64le). Python 3 Module 15 SP6 ppc64le is available on SUSE Linux Enterprise Server 15 SP5 ppc64le, SUSE Linux Enterprise Server for SAP Applications 15 SP5 ppc64le.' (422)
eTLB5-67-

Seems a product bug on aarch64 and ppc64le, We can double check it with developer :)

Actions #6

Updated by amanzini 12 months ago · Edited

  • Status changed from In Progress to Feedback
  • % Done changed from 0 to 30

on ppc64le platform, Python3 module can not be activated, error message refers to 15 SP5 ?

https://openqa.suse.de/tests/13223290

Activating sle-module-python3 15.6 ppc64le ...
Error: Registration server returned 'The product you are attempting to activate (Python 3 Module 15 SP6 ppc64le) is not available on your system's base product (SUSE Linux Enterprise Server 15 SP6 ppc64le). Python 3 Module 15 SP6 ppc64le is available on SUSE Linux Enterprise Server 15 SP5 ppc64le, SUSE Linux Enterprise Server for SAP Applications 15 SP5 ppc64le.' (422)
eTLB5-67-

opened https://bugzilla.suse.com/show_bug.cgi?id=1218683

on aarch64, https://openqa.suse.de/tests/13223289/logfile?filename=serial_terminal.txt

0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1358 ====================================
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1362 nothing provides 'qemu-kvm' needed by the to be installed systemd-testsuite-254.5-150600.2.4.aarch64
0| 2024-01-10 02:37:43 <1> susetest(4532) [zypp::solver] SATResolver.cc(problems):1363 ------------------------------------

filed https://bugzilla.suse.com/show_bug.cgi?id=1218684

Actions #7

Updated by amanzini 11 months ago

  • Status changed from Feedback to In Progress
Actions #8

Updated by amanzini 11 months ago · Edited

I'd ask your opinion on how to handle failures on this kind of systemd test: https://openqa.suse.de/tests/13225554
basically the test cases run in openQA but they are not developed from us, as they come from upstream testsuite and dinamically scheduled from the initial module:

    foreach my $test (@schedule) {
        my $args = OpenQA::Test::RunArgs->new(test => $test, dir => $testdir, make_opts => $test_opts);
        autotest::loadtest('tests/systemd_testsuite/runner.pm', name => $test, run_args => $args);
    }   

should we open a bugzilla for each failing test case ? Contact the developers, try to fix ourself ? Most of them just end due to timeouts like

[   67.521547] testsuite-17.sh[520]: + echo 'Subtest /usr/lib/systemd/tests/testdata/units/testsuite-17.02.sh failed'
[   67.541776] testsuite-17.sh[520]: Subtest /usr/lib/systemd/tests/testdata/units/testsuite-17.02.sh failed
[   67.552700] testsuite-17.sh[520]: + return 1
[   79.177547][    T1] systemd-shutdown[1]: Waiting for process: 491 (systemd-udevd), 772 ((udev-worker))
[  101.240673][  T491] systemd-udevd[491]: foobar: Worker [772] processing SEQNUM=1993 is taking a long time
[  132.281233][  T772] (udev-worker)[772]: hoge: Failed to rename network interface 4 from 'foobar' to 'hoge': File exists
[  132.298053][  T772] (udev-worker)[772]: foobar: Failed to process device, ignoring: File exists

seems they are running a qemu VM for each test case (so at the end it's 3 layers of nested virtualization) ...

Actions #9

Updated by amanzini 11 months ago · Edited

  • % Done changed from 30 to 40

seems like some tests are always failing on SLE, because package needs some modifications. Example run: https://openqa.suse.de/tests/13259003

After discussing with Thomas Blume, we came up with the idea to softfail some specific tests.
sample VR: https://openqa.suse.de/tests/13264222

the one that are safe to softfail are controlled via the variable SYSTEMD_SOFTFAIL in the testsuite settings.
Currently it's
SYSTEMD_SOFTFAIL=(13-NSPAWN|16-EXTEND-TIMEOUT|17-UDEV|24-CRYPTSETUP|29-PORTABLE|43-PRIVATEUSER-UNPRIV|50-DISSECT|55-OOMD|62-RESTRICT-IFACES|65-ANALYZE|67-INTEGRITY|70-TPM2|82-SOFTREBOOT)

Actions #11

Updated by amanzini 11 months ago

  • Status changed from In Progress to Feedback
Actions #12

Updated by amanzini 11 months ago

proposed a way to tackle the sub-tests failure; either excluding them or via softfail. After merge, we need only to schedule this module

Actions #14

Updated by amanzini 11 months ago

  • Status changed from Feedback to Resolved
Actions #17

Updated by openqa_review 10 months ago

  • Status changed from Resolved to Feedback

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: upstream_systemd_tests
https://openqa.suse.de/tests/13493808

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions #18

Updated by amanzini 10 months ago

  • Assignee deleted (amanzini)

removing assignment due to end of squad rotation

Actions #19

Updated by openqa_review 9 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: upstream_systemd_tests
https://openqa.suse.de/tests/13647391

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions #20

Updated by openqa_review 8 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: upstream_systemd_tests
https://openqa.suse.de/tests/13958836

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 68 days if nothing changes in this ticket.

Actions #21

Updated by dzedro 6 months ago

  • Status changed from Feedback to Resolved
  • % Done changed from 40 to 100
Actions

Also available in: Atom PDF