QA » openQA Project » openQA Tests

There seems to be no "python" in https://openqa.suse.de/tests/2492870#step/userspace_apparmor/48 . That was unexpected :D

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6878 to fix first before we could switch the scenarios to the different run method.

For direct comparison:

• SLE15SP1 userspace regression test on production: https://openqa.suse.de/tests/2488712#
• New approach based on QA_TESTSET including above PR: https://openqa.suse.de/tests/2492918
• QAM equivalent to above with QA_TESTSET: https://openqa.suse.de/tests/2492722

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6879 is a more experimental PR, going further

both mentioned PRs merged. Maintenance tests still look very green, e.g. https://openqa.suse.de/tests/overview?distri=sle&version=12-SP3&build=20190301-1&groupid=108 . Further examples: mau-qa_acceptance_fs_stress and mau-qa_userspace_bind

Changing a limited set of test suites, e.g. "qa_userspace_apparmor":

-VIRTIO_CONSOLE=0
-MAX_JOB_TIME=14400
-QA_HEAD_REPO=http://dist.nue.suse.com/ibs/QA:/Head/SLE-%VERSION%
-QA_TESTSUITE=apparmor
+QA_TESTSET=userspace_apparmor

Also adopted "qa_userspace_apparmor-profiles".

Let's wait for a new build.

One side-effect is #48653.

• handle result of https://openqa.suse.de/tests/2513547#step/userspace_apparmor/15 -> "# Test died: No QA_HEAD_REPO specified! at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/qa_automation/qa_run.pm line 84." -> I guess I still need QA_HEAD_REPO (for now). Readded QA_HEAD_REPO=http://dist.nue.suse.com/ibs/QA:/Head/SLE-%VERSION% to both testsuites changed in #44138#note-13

for i in 2513547 2513388 2513180 2513222 2513567 ; do openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i QA_HEAD_REPO=http://dist.nue.suse.com/ibs/QA:/Head/SLE-15-SP1 ; done

Created job #2518055: sle-15-SP1-Installer-DVD-x86_64-Build181.3-qa_userspace_apparmor@64bit -> http://openqa.suse.de/t2518055 -> now failing as expected in the same bug as in before
Created job #2518056: sle-15-SP1-Installer-DVD-aarch64-Build181.3-qa_userspace_apparmor-profiles@aarch64-virtio -> http://openqa.suse.de/t2518056
Created job #2518057: sle-15-SP1-Installer-DVD-ppc64le-Build181.3-qa_userspace_apparmor-profiles@ppc64le -> http://openqa.suse.de/t2518057
Created job #2518058: sle-15-SP1-Installer-DVD-s390x-Build181.3-qa_userspace_apparmor-profiles@s390x-kvm-sle12 -> http://openqa.suse.de/t2518058
Created job #2518059: sle-15-SP1-Installer-DVD-x86_64-Build181.3-qa_userspace_apparmor-profiles@64bit -> http://openqa.suse.de/t2518059 -> failed weirdly in an intermediate step, retrying: https://openqa.suse.de/tests/2518080 . Failed the same however I have seen the same error in https://openqa.suse.de/tests/2518055#step/userspace_apparmor/30 without any bad effect so the error must be something different. https://openqa.suse.de/tests/2518080/file/serial_terminal.txt shows

# /usr/share/qa/qaset/bin/junit_xml_gen.py -n 'regression' -d -o /tmp/junit.xml /var/log/qaset ; echo _dMLG-$?-
[/usr/share/qa/qaset/bin/junit_xml_gen.py]WARNING: No submission info for testsuite apparmor-profiles
[/usr/share/qa/qaset/bin/junit_xml_gen.py]WARNING: No log dir inside submission log of testsuite apparmor-profiles
[/usr/share/qa/qaset/bin/junit_xml_gen.py]DEBUG: Parsing testsuite iostat: iostat
Traceback (most recent call last):
  File "/usr/share/qa/qaset/bin/junit_xml_gen.py", line 512, in <module>
    dc.collect_log()
  File "/usr/share/qa/qaset/bin/junit_xml_gen.py", line 426, in collect_log
    shutil.rmtree(ts_dir)
  File "/usr/lib64/python2.7/shutil.py", line 253, in rmtree
    onerror(os.listdir, path, sys.exc_info())
  File "/usr/lib64/python2.7/shutil.py", line 251, in rmtree
    names = os.listdir(path)
OSError: [Errno 20] Not a directory: '/tmp/junit_generator-66651/iostat'

so for now I am changing "apparmor-profiles" back to the original.

I suggest the following next step:

• Test all scenarios in "userspace" with QA_TESTSET=userspace-<name> and VIRTIO_CONSOLE=1 instead of QA_TESTSUITE=<name>
• understand why some do not work, e.g. as above "apparmor-profiles"
• @waitfor Ask others if QA_TESTSUITE has any benefit over QA_TESTSET -> propose deletion and ask in PR -> @waitfor https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6955
• @waitfor ^{^} Change all/more scenarios to QA_TESTSET instead

Trying to clone all scenarios from latest 15sp1 build with testset:

for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output  '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace-${name}" TEST=qa_userspace_${name}-testset-poo44138 _GROUP="Test Development: SLE 15" ;done

-> https://openqa.suse.de/tests/overview?distri=sle&build=189.1&version=15-SP1&groupid=96

Hm, they all failed. Probably because I used "userspace-${name}" instead of "userspace_${name}". Also the results are actually hard to see in the development job group, trying again in it's own (implicit) build:

for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output  '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace_${name}" TEST=qa_userspace_${name}-testset-poo44138 _GROUP=0 ;done

-> Created job #2537657: sle-15-SP1-Installer-DVD-aarch64-Build189.1-qa_userspace_apache2_mod_perl@aarch64-virtio -> http://openqa.suse.de/t2537657

hm, that was also a bad idea as this is now the same build but without any group -> https://openqa.suse.de/tests/overview?version=15-SP1&build=189.1&distri=sle

As the tests do not take that long I deleted some still scheduled ones:

for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 state=scheduled arch=s390x | jq --raw-output --join-output  '.jobs[] | (.id," ")') ; do openqa-client --host openqa.suse.de jobs/$i delete ; done

and I retriggered with a custom build:

for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output  '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace_${name}" TEST=qa_userspace_${name}-testset-poo44138 BUILD=189.1@poo44138 _GROUP=0 ;done

Created job #2537763: sle-15-SP1-Installer-DVD-aarch64-Build189.1-qa_userspace_apache2_mod_perl@aarch64-virtio -> http://openqa.suse.de/t2537763 -> https://openqa.suse.de/tests/overview?version=15-SP1&build=189.1%40poo44138&distri=sle

The following testsuites seem to be problematic:

• apache2-mod_perl -> #49106
• apparmor-profiles (already mentioned above) -> #49106
• sharutils (failed on s390x) -> #49112
• php (should have been php7) -> #49115

the following fail but should be fine:

• nfs_v4 (was scheduled as "null", should be checked specifically)
• openssh failed the same as in before https://openqa.suse.de/tests/2533808
• systemd fails the same as in before https://openqa.suse.de/tests/2536713

so I changed all the testsuites except the "problematic" ones as mentioned above to use "TESTSET" instead.

The only part left for this commit is I should await a next official SLE15SP1 build with the according test results in the userspace job group.