Project

General

Profile

Actions

action #127355

closed

[security][15sp4][15sp5][qu] test fails in oscap_xccdf_eval#1

Added by amanzini 12 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
-
Start date:
2023-04-06
Due date:
% Done:

100%

Estimated time:
24.00 h
Difficulty:
Tags:

Description

Observation

openQA test in scenario sle-15-SP5-Online-ppc64le-stig_hmc@ppc64le-hmc-single-disk fails in
oscap_xccdf_eval#1

Test suite description

Testsuite maintained at https://gitlab.suse.de/qe-security/osd-sle15-security.

Reproducible

Fails since (at least) Build 24.1

Expected result

Last good: 21.1 (or more recent)

Further details

Always latest result in this scenario: latest


Related issues 1 (0 open1 closed)

Related to openQA Tests - action #129718: [security][qu][ppc64le] test fails in oscap_xccdf_eval#1Rejected2023-05-23

Actions
Actions #1

Updated by emiler 12 months ago

  • Assignee set to emiler
Actions #2

Updated by emiler 12 months ago

Looks like it is out of memory despite having QEMURAM=24576. I will experiment with different values.

Actions #4

Updated by emiler 12 months ago

I am getting different failures with re-runs:

Memory seems to be enough in both cases, which is weird, because the first has the original amount of memory.

Actions #5

Updated by emiler 12 months ago

Oh, I seem to have been looking at older builds. The low memory was fixed by https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/83/diffs. Latest build 88.1 here (original link from description is dead): https://openqa.suse.de/tests/10869446#step/oscap_xccdf_eval#1/159

Actions #6

Updated by emiler 12 months ago

Investigation by Viktor (thanks for the help) revealed several issues:

Some runs are executed on LPAR -id 6 (selinux enabled) and my re-runs on LPAR -id 12 (fips enabled), therefore resulting in different errors. We need to reach out to PPC team to make them run stable.

Re-runs revealed an issue with missing expected results for ppc in https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/1dd47e5c6d5f47d9377cc4d4afc4bcd30ab7a963/tests/security/oscap_stig/oscap_xccdf_eval.pm# (sourced from https://openqa.suse.de/tests/10912059#step/oscap_xccdf_eval#1/7). For example:

if (is_ppc64le) {
    $n_passed_rules = 211;
    $n_failed_rules = 4;
    @eval_match = (
        'content_rule_partition_for_var_log_audit',
        'content_rule_smartcard_pam_enabled',
        'content_rule_grub2_uefi_password',
        'content_rule_no_files_unowned_by_user');
}
Actions #7

Updated by tjyrinki_suse 12 months ago

  • Estimated time set to 24.00 h
Actions #8

Updated by emiler 12 months ago

This issue https://openqa.suse.de/tests/10869446#step/oscap_xccdf_eval#1/159 is caused by https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/Utils/Logging.pm#L265. Line-breaks in commands are not supported except in script_output(). This is apparently being fixed by Pavel Dostál. I'll work on the other issue mentioned above.

Actions #9

Updated by tjyrinki_suse 11 months ago

  • Status changed from New to Workable
Actions #10

Updated by openqa_review 11 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11049384#step/oscap_xccdf_eval#1/1

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions #11

Updated by emiler 11 months ago

  • Assignee deleted (emiler)
Actions #12

Updated by openqa_review 10 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions #13

Updated by amanzini 10 months ago

  • Related to action #129718: [security][qu][ppc64le] test fails in oscap_xccdf_eval#1 added
Actions #14

Updated by pstivanin 10 months ago

  • Subject changed from [security][15sp5]test fails in oscap_xccdf_eval#1 to [security][15sp5][ppc64le] test fails in oscap_xccdf_eval#1
Actions #15

Updated by openqa_review 10 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions #16

Updated by FSzekely 9 months ago

I'm afraid the same issue hit these tests too and it's not only on ppc64:
https://openqa.suse.de/tests/11440768
https://openqa.suse.de/tests/11443583

Actions #17

Updated by FSzekely 9 months ago

  • Subject changed from [security][15sp5][ppc64le] test fails in oscap_xccdf_eval#1 to [security][15sp5] test fails in oscap_xccdf_eval#1
Actions #18

Updated by FSzekely 9 months ago

  • Subject changed from [security][15sp5] test fails in oscap_xccdf_eval#1 to [security][15sp4][15sp5] test fails in oscap_xccdf_eval#1
Actions #19

Updated by pstivanin 9 months ago

  • Subject changed from [security][15sp4][15sp5] test fails in oscap_xccdf_eval#1 to [security][15sp4][15sp5][qu] test fails in oscap_xccdf_eval#1
Actions #20

Updated by pstivanin 9 months ago

Actions #22

Updated by pstivanin 9 months ago

  • Status changed from Workable to In Progress
  • Assignee set to pstivanin
  • % Done changed from 0 to 70

this fixes the issue on s390x and x86_64: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17348

now I'll look into the aarch64 timeout issue

Actions #23

Updated by dzedro 9 months ago

Some timeouts need to be increased for aarch64 https://openqa.suse.de/tests/11498864#step/oscap_xccdf_remediate/4 or use TIMEOUT_SCALE
IMO for maintenance if test does not require snapshots then it should be disabled with QEMU_DISABLE_SNAPSHOTS=1 especially on slow aarch64.
https://openqa.suse.de/tests/11499501 15-SP4
https://openqa.suse.de/tests/11499514 15-SP5

Actions #24

Updated by pstivanin 9 months ago

hello, thanks a lot Jozef :) let me try with that option (which I expect to work since the error is about snapshosts):

[2023-07-04T05:14:56.720292+02:00] [debug] [pid:433] ||| finished oscap_xccdf_remediate security/oscap_stig (runtime: 649 s)
[2023-07-04T05:14:56.723777+02:00] [debug] [pid:433] Creating a VM snapshot lastgood
[2023-07-04T05:14:56.726914+02:00] [debug] [pid:537] Saving snapshot (Current VM state is running)
...
[2023-07-04T05:18:57.042107+02:00] [debug] [pid:537] EVENT {"data":{"status":"cancelling"},"event":"MIGRATION","timestamp":{"microseconds":41692,"seconds":1688440737}}
[2023-07-04T05:18:57.042778+02:00] [info] [pid:537] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
  Migrate to file failed, it has been running for more than 240 seconds at /usr/lib/os-autoinst/backend/qemu.pm line 226.
[2023-07-04T05:18:57.043894+02:00] [info] [pid:537] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
Actions #25

Updated by pstivanin 9 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 70 to 100
Actions #26

Updated by openqa_review 8 months ago

  • Status changed from Resolved to Feedback

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions #27

Updated by pstivanin 8 months ago

  • Status changed from Feedback to Resolved
Actions #28

Updated by openqa_review 8 months ago

  • Status changed from Resolved to Feedback

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
  3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions #29

Updated by pstivanin 8 months ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF