QA » openQA Project » openQA Tests

phase 1 tips:
These tips based on developer's comments.

1. [2C- Starting verifying open network ports (22 25 80 111 443)...[0C Q: I'm surprised to see 80 and 443 in there. Is a webserver installed? A: After investigation 443/80 not opened, products have no issue not sure why lynis listed the ports here.
2. create a newbaseline with apache installed
3. only check the approved baseline if the sections are the same For new sections will check the current outputs only
4. handle duplicated sections e.g., system tools/[+] File systems
5. Are you checking the Hardening index : 90 [################## ] Hardening index : 90 [################## ] numerical value? If not that should be added A: done
6. I had one idea while checking this. Currently lynis runs this check: [2C- Starting dbus policy check...[28C this will likely change rather frequently and we have good mechanisms on our side to tackle this. It might make sense to filter this out A: done

phase 2 tips:

1. Support on other architectures
2. Support gnome img
3. dynamically reload

Test case design tips:
It includes phase 1 and phase 2, see above comments for details.

Some explanations:

1. The dynamical modules are named after section name ([+]_*), the section flag is "[+] "
   There are 51 sections atm.
   Test modules are generated dynamically for easier openQA review (openQA webUI is very slow most time)
   We can just scroll mouse find the section interested and put cursor on the "box" or click "left" key we can get the info very fast.

2. There are still some exceptions are handled. e.g., some sections are with same name
   e,g,, "[+] system tools / [+] File systems..."

3. Baseline file can be set by openQA "Settings" LYNIS_BASELINE_FILE
   The name is constructed by: baseline-$cmds-$product-$arch-$build-$mode

   e.g., The first/default baseline name is "LYNIS_BASELINE_FILE=baseline-lynis-audit-system-nocolors-sle15sp3-x86_64-snapshot7-textmode"
   So we can build different baselines as needed in future.
   Also the current/new baseline can be downloaded for a new baseline.
   e.g., https://openqa.suse.de/tests/5270882#downloads

4. Mapping rules can be set by openQA "Settings":

   LYNIS_ERROR=ERROR,UNSAFE,WEAK -> openQA fail
   LYNIS_OK=OK,DONE,YES -> openQA pass
   LYNIS_WARNING=WARNING,EXPOSED,NONE,SUGGESTION -> openQA softfail

   Atm I only defined some of the status and ignored others.
   I have checked some lynis outputs and here are the most lynis test status FYI:

   [ OK ]/[ .*FOUND ]/[ YES ]/[ NO ]/[ SUGGESTION ]/[ DIFFERENT ]/[ NONE ]/
   [ DISABLED ]/[ DONE ]/[ ENABLED ]/[ NOT ENABLED ]/[ NOT ACTIVE ]/
   [ NOT RUNNING ]/[ NO RESPONSE ]/[ UNSAFE ]/[ ERROR ]/[ WARNING ]/ ...

5. If a new section is found in current lynis output
   Then, openQA test result will be marked as "softfail" firstly. It mean new baseline file needs to be added/used to test code, meanwhile current lynis output will be checked also.

   e.g. https://openqa.suse.de/tests/5270882#step/[+]_Initializing_program/3
   (We can revise baseline file on purpose for generating a softfail)

6. If an old section
   Then baseline and current files will be checked/compared.

7. Each sections' outputs can be checked by "FYI baseline content" and "FYI current content" openQA "box"

8. The lynis tool is located in "PackageHub" so lynis test case is only valid after Beta phase
   (PackageHub will not be ready/available before Beta)

phase 1 PR sent out: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/11667

The first run of phase 1 in openQA is good: https://openqa.suse.de/tests/5398937

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lynis
https://openqa.suse.de/tests/5462211

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.opensuse.org/tests/1682855

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lynis_textmode
https://openqa.opensuse.org/tests/1682853

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.opensuse.org/tests/1697491

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.opensuse.org/tests/1716141

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.suse.de/tests/5991414

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.suse.de/tests/5991414

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lynis_gnome
https://openqa.suse.de/tests/5991414

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

okurz wrote:

@llzhao if you can see this message then you should have seen the multiple reminder comments by openqa_review as well, see above

This soft fail should be tracked by bsc#1185942 on sles15sp3 187.1 in fact as the soft fail found a product bug.
There is no issue need to be handled atm.
This poo should be resolved.

We have another 'poo#91383 - [sle][security][sle15sp4] tracker poo for "Lynis test cases softfails in OpenQA' to track new lynis issues.
It is really a long story why we use this way/poo to track lynis softfail (lynis baselines change often, we use a tracker poo in code and will open new sub poo to update baselines if needed)

Thanks for the explanation Oliver.
I found the root cause: The run of 187.1 did not use the new testing code so the soft fail tag still used "poo#78224".
I did rerun for all the 8 test cases, the tag is right now:
https://openqa.suse.de/tests/6305540#step/18_[+]_Software:_firewalls/10
(# Soft Failure:
poo#91383, found 1 [ WARNING ] in current output)
But the arm work has issue atm, I will check later.

The rerun on arm is done:
https://openqa.suse.de/tests/6305532#step/18_[+]_Software:_firewalls/10
Then next time tracker "poo#91383" will be automatically updated by openqa_review.
Let's keep the status with "Feedback" and check 2 weeks later.