action #127355
closed[security][15sp4][15sp5][qu] test fails in oscap_xccdf_eval#1
Added by amanzini about 1 year ago. Updated 9 months ago.
100%
Description
Observation¶
openQA test in scenario sle-15-SP5-Online-ppc64le-stig_hmc@ppc64le-hmc-single-disk fails in
oscap_xccdf_eval#1
Test suite description¶
Testsuite maintained at https://gitlab.suse.de/qe-security/osd-sle15-security.
Reproducible¶
Fails since (at least) Build 24.1
Expected result¶
Last good: 21.1 (or more recent)
Further details¶
Always latest result in this scenario: latest
Updated by emiler about 1 year ago
Looks like it is out of memory despite having QEMURAM=24576
. I will experiment with different values.
Updated by emiler about 1 year ago
Updated by emiler about 1 year ago
I am getting different failures with re-runs:
- https://openqa.suse.de/tests/10899081#step/oscap_xccdf_eval#1/7
- https://openqa.suse.de/tests/10899086#step/oscap_xccdf_eval#1/7
Memory seems to be enough in both cases, which is weird, because the first has the original amount of memory.
Updated by emiler about 1 year ago
Oh, I seem to have been looking at older builds. The low memory was fixed by https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/83/diffs. Latest build 88.1 here (original link from description is dead): https://openqa.suse.de/tests/10869446#step/oscap_xccdf_eval#1/159
Updated by emiler about 1 year ago
Investigation by Viktor (thanks for the help) revealed several issues:
Some runs are executed on LPAR -id 6 (selinux enabled) and my re-runs on LPAR -id 12 (fips enabled), therefore resulting in different errors. We need to reach out to PPC team to make them run stable.
Re-runs revealed an issue with missing expected results for ppc in https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/1dd47e5c6d5f47d9377cc4d4afc4bcd30ab7a963/tests/security/oscap_stig/oscap_xccdf_eval.pm# (sourced from https://openqa.suse.de/tests/10912059#step/oscap_xccdf_eval#1/7). For example:
if (is_ppc64le) {
$n_passed_rules = 211;
$n_failed_rules = 4;
@eval_match = (
'content_rule_partition_for_var_log_audit',
'content_rule_smartcard_pam_enabled',
'content_rule_grub2_uefi_password',
'content_rule_no_files_unowned_by_user');
}
Updated by emiler about 1 year ago
This issue https://openqa.suse.de/tests/10869446#step/oscap_xccdf_eval#1/159 is caused by https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/Utils/Logging.pm#L265. Line-breaks in commands are not supported except in script_output()
. This is apparently being fixed by Pavel Dostál. I'll work on the other issue mentioned above.
Updated by openqa_review 12 months ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11049384#step/oscap_xccdf_eval#1/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by openqa_review 11 months ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by amanzini 11 months ago
- Related to action #129718: [security][qu][ppc64le] test fails in oscap_xccdf_eval#1 added
Updated by openqa_review 11 months ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by FSzekely 10 months ago
I'm afraid the same issue hit these tests too and it's not only on ppc64:
https://openqa.suse.de/tests/11440768
https://openqa.suse.de/tests/11443583
Updated by pstivanin 10 months ago
happens also on QU (https://progress.opensuse.org/issues/129718)
Updated by pstivanin 10 months ago
failed also today on 15-sp4, 15-sp5, all archs:
15sp4:
- https://openqa.suse.de/tests/11495561
- https://openqa.suse.de/tests/11495569
- https://openqa.suse.de/tests/11495552
15sp5
Updated by pstivanin 10 months ago
- Status changed from Workable to In Progress
- Assignee set to pstivanin
- % Done changed from 0 to 70
this fixes the issue on s390x and x86_64: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17348
now I'll look into the aarch64 timeout issue
Updated by dzedro 10 months ago
Some timeouts need to be increased for aarch64 https://openqa.suse.de/tests/11498864#step/oscap_xccdf_remediate/4 or use TIMEOUT_SCALE
IMO for maintenance if test does not require snapshots then it should be disabled with QEMU_DISABLE_SNAPSHOTS=1 especially on slow aarch64.
https://openqa.suse.de/tests/11499501 15-SP4
https://openqa.suse.de/tests/11499514 15-SP5
Updated by pstivanin 10 months ago
hello, thanks a lot Jozef :) let me try with that option (which I expect to work since the error is about snapshosts):
[2023-07-04T05:14:56.720292+02:00] [debug] [pid:433] ||| finished oscap_xccdf_remediate security/oscap_stig (runtime: 649 s)
[2023-07-04T05:14:56.723777+02:00] [debug] [pid:433] Creating a VM snapshot lastgood
[2023-07-04T05:14:56.726914+02:00] [debug] [pid:537] Saving snapshot (Current VM state is running)
...
[2023-07-04T05:18:57.042107+02:00] [debug] [pid:537] EVENT {"data":{"status":"cancelling"},"event":"MIGRATION","timestamp":{"microseconds":41692,"seconds":1688440737}}
[2023-07-04T05:18:57.042778+02:00] [info] [pid:537] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
Migrate to file failed, it has been running for more than 240 seconds at /usr/lib/os-autoinst/backend/qemu.pm line 226.
[2023-07-04T05:18:57.043894+02:00] [info] [pid:537] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
Updated by pstivanin 10 months ago
- Status changed from In Progress to Resolved
- % Done changed from 70 to 100
this fixes aarch64: https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/134
Updated by openqa_review 10 months ago
- Status changed from Resolved to Feedback
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by openqa_review 9 months ago
- Status changed from Resolved to Feedback
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.