action #121747
closed[security] Migrate security testing to using settings in job group and yaml schedules and cleanup code
100%
Description
Currently security tests are not using the more modern openQA ways of scheduling tests, making it more cumbersome to adapt tests to new products or make changes.
We should move from using hard coded execution logic in perl code to yaml definitions.
Acceptance Criteria¶
- Remove the following functions from main_common_pm: load_security_* if possible
- Check if our job groups define extra parameters that could be dropped, or moved to common/nested settings for easier manageability (see opensuse-jobgroups/job_groups/alp.yaml for an example)
Updated by pstivanin over 1 year ago
- Status changed from New to In Progress
- Assignee set to pstivanin
Updated by pstivanin over 1 year ago
- % Done changed from 0 to 30
clean up for maintenance: https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/36
Updated by pstivanin over 1 year ago
- % Done changed from 30 to 50
MR for devel jobgroup: https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/37
Updated by pstivanin over 1 year ago
- % Done changed from 50 to 60
MR for QU jobgroup: https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/38
Updated by pstivanin over 1 year ago
Updated by pstivanin over 1 year ago
- % Done changed from 60 to 70
Since it's not possible to migrate to yaml code that contains &&, ||, etc, we cannot migrate everything from main_common. For example, code like the following:
if (get_var("SECURITY_TEST") =~ /^crypt_/ && !is_opensuse && (get_var("BETA") || check_var("FLAVOR", "Online-QR")));
or
if (get_var("FIPS_ENABLED") && get_var("JEOS"));
must remain in main_common.
Updated by pstivanin over 1 year ago
Since load_security_console_prepare contains non migratable code, the following tests cannot be ported to yaml:
load_security_tests_crypt_core
load_security_tests_crypt_web
load_security_tests_crypt_kernel
load_security_tests_crypt_x11
load_security_tests_crypt_firefox
load_security_tests_crypt_openjdk
load_security_tests_crypt_tool
load_security_tests_crypt_libtool
load_security_tests_ipsec
load_security_tests_mmtest
load_security_tests_apparmor
load_security_tests_apparmor_profile
load_security_tests_yast2_apparmor
load_security_tests_yast2_users
load_security_tests_lynis
load_security_tests_openscap
load_security_tests_cc_audit_test
load_security_tests_check_kernel_config
load_security_tests_pam
load_security_tests_create_swtpm_hdd
load_security_tests_swtpm
load_security_tests_grub_auth
load_security_tests_tpm2
load_security_tests_fips_setup
load_security_tests_cc_audit_remote_libvirt
load_security_tests_mok_enroll
the following tests, though, seem to be migratable to yaml:
load_security_tests_crypt_krb5kdc
load_security_tests_crypt_krb5server
load_security_tests_crypt_krb5client
load_security_tests_selinux
load_security_tests_ima_measurement
load_security_tests_ima_appraisal
load_security_tests_evm_protection
load_security_tests_system_check
Updated by pstivanin over 1 year ago
- % Done changed from 70 to 80
The following tests have been ported to yaml:
load_security_tests_crypt_krb5kdc
load_security_tests_crypt_krb5server
load_security_tests_crypt_krb5client
load_security_tests_selinux
load_security_tests_ima_measurement
load_security_tests_ima_appraisal
load_security_tests_evm_protection
load_security_tests_system_check
Only things left are to adjust main_common, check if opensuse is using those and update the jsons to point the new yaml.
Updated by pstivanin over 1 year ago
- % Done changed from 80 to 90
Test suites on OSD: updated
Test suites on O3: updated
PRs:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/16109
https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/41
Updated by pstivanin over 1 year ago
- Status changed from In Progress to Resolved
- % Done changed from 90 to 100
PRs merged.
Updated by punkioudi over 1 year ago
- Status changed from Resolved to In Progress
Sorry for the noise!
In the latest 15-SP5 runs there were some failures because of "wrong" expected qcow2 images:
https://openqa.suse.de/tests/10186640
https://openqa.suse.de/tests/10187557
https://openqa.suse.de/tests/10186338
https://openqa.suse.de/tests/10187559
https://openqa.suse.de/tests/10185735
https://openqa.suse.de/tests/10185731
I re-open this ticket as they seem to be part of this cleanup, if not I can open a new one :)
Updated by pstivanin over 1 year ago
- % Done changed from 100 to 80
Thanks Anna, I'll look into it :)
Updated by pstivanin over 1 year ago
- % Done changed from 80 to 100
this mr should fix those issues: https://gitlab.suse.de/qe-security/osd-sle15-security/-/merge_requests/45
Updated by pstivanin over 1 year ago
- Status changed from In Progress to Resolved
PR has been merged. With the next run it will pick up the changes and everything should be fine.
Closing the ticket for now.
Updated by openqa_review over 1 year ago
- Status changed from Resolved to Feedback
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: security_tpm2_keylime_uefi@uefi
https://openqa.suse.de/tests/10256004#step/boot_to_desktop/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by pstivanin over 1 year ago
- Status changed from Feedback to Resolved