QA » openQA Project » openQA Tests

Looks like it is out of memory despite having QEMURAM=24576. I will experiment with different values.

Related:

• https://progress.opensuse.org/issues/126713
• https://progress.opensuse.org/issues/115907

I am getting different failures with re-runs:

• https://openqa.suse.de/tests/10899081#step/oscap_xccdf_eval#1/7
• https://openqa.suse.de/tests/10899086#step/oscap_xccdf_eval#1/7

Memory seems to be enough in both cases, which is weird, because the first has the original amount of memory.

Oh, I seem to have been looking at older builds. The low memory was fixed by https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/83/diffs. Latest build 88.1 here (original link from description is dead): https://openqa.suse.de/tests/10869446#step/oscap_xccdf_eval#1/159

Investigation by Viktor (thanks for the help) revealed several issues:

Some runs are executed on LPAR -id 6 (selinux enabled) and my re-runs on LPAR -id 12 (fips enabled), therefore resulting in different errors. We need to reach out to PPC team to make them run stable.

Re-runs revealed an issue with missing expected results for ppc in https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/1dd47e5c6d5f47d9377cc4d4afc4bcd30ab7a963/tests/security/oscap_stig/oscap_xccdf_eval.pm# (sourced from https://openqa.suse.de/tests/10912059#step/oscap_xccdf_eval#1/7). For example:

if (is_ppc64le) {
    $n_passed_rules = 211;
    $n_failed_rules = 4;
    @eval_match = (
        'content_rule_partition_for_var_log_audit',
        'content_rule_smartcard_pam_enabled',
        'content_rule_grub2_uefi_password',
        'content_rule_no_files_unowned_by_user');
}

This issue https://openqa.suse.de/tests/10869446#step/oscap_xccdf_eval#1/159 is caused by https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/Utils/Logging.pm#L265. Line-breaks in commands are not supported except in script_output(). This is apparently being fixed by Pavel Dostál. I'll work on the other issue mentioned above.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11049384#step/oscap_xccdf_eval#1/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: stig_hmc
https://openqa.suse.de/tests/11162641#step/oscap_xccdf_eval#1/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

I'm afraid the same issue hit these tests too and it's not only on ppc64:
https://openqa.suse.de/tests/11440768
https://openqa.suse.de/tests/11443583

happens also on QU (https://progress.opensuse.org/issues/129718)

failed also today on 15-sp4, 15-sp5, all archs:

15sp4:

• https://openqa.suse.de/tests/11495561
• https://openqa.suse.de/tests/11495569
• https://openqa.suse.de/tests/11495552

15sp5

• https://openqa.suse.de/tests/11495638
• https://openqa.suse.de/tests/11495628
• https://openqa.suse.de/tests/11495557

Some timeouts need to be increased for aarch64 https://openqa.suse.de/tests/11498864#step/oscap_xccdf_remediate/4 or use TIMEOUT_SCALE
IMO for maintenance if test does not require snapshots then it should be disabled with QEMU_DISABLE_SNAPSHOTS=1 especially on slow aarch64.
https://openqa.suse.de/tests/11499501 15-SP4
https://openqa.suse.de/tests/11499514 15-SP5

hello, thanks a lot Jozef :) let me try with that option (which I expect to work since the error is about snapshosts):

[2023-07-04T05:14:56.720292+02:00] [debug] [pid:433] ||| finished oscap_xccdf_remediate security/oscap_stig (runtime: 649 s)
[2023-07-04T05:14:56.723777+02:00] [debug] [pid:433] Creating a VM snapshot lastgood
[2023-07-04T05:14:56.726914+02:00] [debug] [pid:537] Saving snapshot (Current VM state is running)
...
[2023-07-04T05:18:57.042107+02:00] [debug] [pid:537] EVENT {"data":{"status":"cancelling"},"event":"MIGRATION","timestamp":{"microseconds":41692,"seconds":1688440737}}
[2023-07-04T05:18:57.042778+02:00] [info] [pid:537] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
  Migrate to file failed, it has been running for more than 240 seconds at /usr/lib/os-autoinst/backend/qemu.pm line 226.
[2023-07-04T05:18:57.043894+02:00] [info] [pid:537] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json