action #78224
closed[sle][security][sle15sp3] Integrate the Lynis scanner into OpenQA
Added by llzhao over 3 years ago. Updated over 2 years ago.
100%
Description
This poo is a parent poo.
Requirements from developer:
One idea would be to include lynis into openQA. This is something we wanted
to do for a long time, but never found the time to do it. So if you have
the time and expertise to include this into openQA this would be great.
We would still use this in our product testing, but having some regular
baseline testing would be interesting. If you're interested have one of the
engineers make themselves familiar with lynis. Lynis offers various output
formats and it probably make sense to check them out and chose one of the
machine readable formats well suited for automation. I think we should
cover these cases:
- Ensure that tests that we pass don't change into negatives
- Ensure that newly added test to lynis don't fail
The engineer will need to figure out a good balance between catching issues
and not being to rigid with the test output. It doesn't make sense to match
on an exact lynis output since this will likely change. But we could e.g.
match the high level result returned at the end of an run, that looked
rather stable to me
links:
https://cisofy.com/lynis/
Updated by llzhao over 3 years ago
- Assignee set to llzhao
- Estimated time set to 160.00 h
- Difficulty set to medium
Updated by llzhao over 3 years ago
phase 1 tips:
These tips based on developer's comments.
- [2C- Starting verifying open network ports (22 25 80 111 443)...[0C Q: I'm surprised to see 80 and 443 in there. Is a webserver installed? A: After investigation 443/80 not opened, products have no issue not sure why lynis listed the ports here.
- create a newbaseline with apache installed
- only check the approved baseline if the sections are the same For new sections will check the current outputs only
- handle duplicated sections e.g., system tools/[+] File systems
- Are you checking the Hardening index : 90 [################## ] Hardening index : 90 [################## ] numerical value? If not that should be added A: done
- I had one idea while checking this. Currently lynis runs this check: [2C- Starting dbus policy check...[28C this will likely change rather frequently and we have good mechanisms on our side to tackle this. It might make sense to filter this out A: done
phase 2 tips:
- Support on other architectures
- Support gnome img
- dynamically reload
Updated by llzhao over 3 years ago
Test case design tips:
It includes phase 1 and phase 2, see above comments for details.
Some explanations:
The dynamical modules are named after section name ([+]_*), the section flag is "[+] "
There are 51 sections atm.
Test modules are generated dynamically for easier openQA review (openQA webUI is very slow most time)
We can just scroll mouse find the section interested and put cursor on the "box" or click "left" key we can get the info very fast.There are still some exceptions are handled. e.g., some sections are with same name
e,g,, "[+] system tools / [+] File systems..."Baseline file can be set by openQA "Settings" LYNIS_BASELINE_FILE
The name is constructed by: baseline-$cmds-$product-$arch-$build-$modee.g., The first/default baseline name is "LYNIS_BASELINE_FILE=baseline-lynis-audit-system-nocolors-sle15sp3-x86_64-snapshot7-textmode"
So we can build different baselines as needed in future.
Also the current/new baseline can be downloaded for a new baseline.
e.g., https://openqa.suse.de/tests/5270882#downloadsMapping rules can be set by openQA "Settings":
LYNIS_ERROR=ERROR,UNSAFE,WEAK -> openQA fail
LYNIS_OK=OK,DONE,YES -> openQA pass
LYNIS_WARNING=WARNING,EXPOSED,NONE,SUGGESTION -> openQA softfailAtm I only defined some of the status and ignored others.
I have checked some lynis outputs and here are the most lynis test status FYI:[ OK ]/[ .*FOUND ]/[ YES ]/[ NO ]/[ SUGGESTION ]/[ DIFFERENT ]/[ NONE ]/
[ DISABLED ]/[ DONE ]/[ ENABLED ]/[ NOT ENABLED ]/[ NOT ACTIVE ]/
[ NOT RUNNING ]/[ NO RESPONSE ]/[ UNSAFE ]/[ ERROR ]/[ WARNING ]/ ...If a new section is found in current lynis output
Then, openQA test result will be marked as "softfail" firstly. It mean new baseline file needs to be added/used to test code, meanwhile current lynis output will be checked also.e.g. https://openqa.suse.de/tests/5270882#step/[+]_Initializing_program/3
(We can revise baseline file on purpose for generating a softfail)If an old section
Then baseline and current files will be checked/compared.Each sections' outputs can be checked by "FYI baseline content" and "FYI current content" openQA "box"
The lynis tool is located in "PackageHub" so lynis test case is only valid after Beta phase
(PackageHub will not be ready/available before Beta)
Updated by llzhao over 3 years ago
- Related to action #88153: [sle][security][sle15sp3] Integrate Lynis into OpenQA - setup env added
Updated by llzhao over 3 years ago
- Related to deleted (action #88153: [sle][security][sle15sp3] Integrate Lynis into OpenQA - setup env )
Updated by llzhao over 3 years ago
- Related to action #88153: [sle][security][sle15sp3] Integrate Lynis into OpenQA - setup env added
Updated by llzhao over 3 years ago
- Related to deleted (action #88153: [sle][security][sle15sp3] Integrate Lynis into OpenQA - setup env )
Updated by llzhao over 3 years ago
phase 1 PR sent out: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/11667
Updated by llzhao about 3 years ago
- Status changed from Feedback to In Progress
Updated by llzhao about 3 years ago
The first run of phase 1 in openQA is good: https://openqa.suse.de/tests/5398937
Updated by openqa_review about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: lynis
https://openqa.suse.de/tests/5462211
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by llzhao about 3 years ago
- Status changed from In Progress to Feedback
Phase 2 PR sent out and merged: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/11998
Updated by okurz about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.opensuse.org/tests/1682855
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by okurz about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: lynis_textmode
https://openqa.opensuse.org/tests/1682853
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by okurz about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.opensuse.org/tests/1697491
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by openqa_review about 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.opensuse.org/tests/1716141
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by okurz almost 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.suse.de/tests/5991414
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by okurz almost 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.suse.de/tests/5991414
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by openqa_review almost 3 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.suse.de/tests/5991414
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released"
- The label in the openQA scenario is removed
Updated by okurz almost 3 years ago
- Status changed from Resolved to Feedback
@llzhao if you can see this message then you should have seen the multiple reminder comments by openqa_review as well, see above
Updated by llzhao almost 3 years ago
okurz wrote:
@llzhao if you can see this message then you should have seen the multiple reminder comments by openqa_review as well, see above
This soft fail should be tracked by bsc#1185942 on sles15sp3 187.1 in fact as the soft fail found a product bug.
There is no issue need to be handled atm.
This poo should be resolved.
We have another 'poo#91383 - [sle][security][sle15sp4] tracker poo for "Lynis test cases softfails in OpenQA' to track new lynis issues.
It is really a long story why we use this way/poo to track lynis softfail (lynis baselines change often, we use a tracker poo in code and will open new sub poo to update baselines if needed)
Updated by okurz almost 3 years ago
- Status changed from Resolved to Feedback
Hi llzhao, thanks for your answer. But we seem to misunderstand: openqa_review simply looks at which ticket is referenced in the test either by bug ref or within the test code with record_soft_failure. In the case of the last referenced job, https://openqa.suse.de/tests/5991414#step/18_[+]_Software:_firewalls/10 explicitly references this ticket so as long as the test code triggers the same, the ticket will receive reminder comments. Simply resolving this ticket does not solve it. I did not find a reference to poo#78224 in the test code anywhere in os-autoinst-distri-opensuse directly, but referenced within the "Tags" of test modules. So I don't know how the test code triggers the reference to the ticket but still this needs to be changed on that side.
Updated by llzhao almost 3 years ago
Thanks for the explanation Oliver.
I found the root cause: The run of 187.1 did not use the new testing code so the soft fail tag still used "poo#78224".
I did rerun for all the 8 test cases, the tag is right now:
https://openqa.suse.de/tests/6305540#step/18_[+]_Software:_firewalls/10
(# Soft Failure:
poo#91383, found 1 [ WARNING ] in current output)
But the arm work has issue atm, I will check later.
Updated by llzhao almost 3 years ago
The rerun on arm is done:
https://openqa.suse.de/tests/6305532#step/18_[+]_Software:_firewalls/10
Then next time tracker "poo#91383" will be automatically updated by openqa_review.
Let's keep the status with "Feedback" and check 2 weeks later.
Updated by llzhao over 2 years ago
- Status changed from Feedback to Resolved
There is no "This is an autogenerated message for openQA integration by the openqa_review script:" for more than 1 month, so mark it as resolved.