Project

General

Profile

action #44138

[functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheck

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
SUSE QA tests - Milestone 24
Start date:
2018-11-21
Due date:
% Done:

0%

Estimated time:
Difficulty:
Duration:

Description

Observation

openQA test in scenario sle-15-SP1-Installer-DVD-x86_64-qa_userspace_openssl@64bit fails in
execute_test_run
and then the details show the step https://openqa.suse.de/tests/2269611#step/1_openssl/83 failing but not showing any good feedback what is broken there.
However, there are QAM tests, like https://openqa.suse.de/tests/2268466# which look equivalent but have a different way of outputting. Can we achieve the same?

Expected result

The failed steps or modules should provide log output which points to the likely errors

Further details

Always latest result in this scenario: latest


Related issues

Related to openQA Tests - action #44843: [functional][u][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module)Workable2018-12-13

Related to openQA Tests - action #48653: [functional][u] test fails in kernel_multipath and mau-qa_kernel_kexec on SLE-12 SP3 and newerResolved2019-03-04

Blocks openQA Tests - action #44828: [sle][functional][u][userspace] test fails in execute_test_run (test_verify) - need to update test code and report new test resultRejected2018-12-06

Blocks openQA Tests - action #44882: [functional][sle][userspace][u]test fails in execute_test_run - almost all php tests failedRejected2018-12-07

Blocks openQA Tests - action #46490: [functional][u][sporadic] test fails in execute_test_run: char gets double-typedResolved2019-01-22

Precedes openQA Tests - action #44195: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failedRejected2018-11-22

Copied to openQA Tests - action #49118: [functional][u][userspace] Investigate if QA_TESTSUITE has any benefits over QA_TESTSETRejected2018-11-21

History

#2 Updated by okurz over 1 year ago

  • Parent task set to #34663

#4 Updated by okurz over 1 year ago

  • Related to action #44843: [functional][u][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module) added

#5 Updated by okurz over 1 year ago

  • Target version changed from Milestone 22 to Milestone 24

#6 Updated by okurz over 1 year ago

  • Status changed from New to Feedback
  • Assignee set to okurz

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4132 with title "qa_automation: add new design for QA:HEAD tests" added the use of the test variable QA_TESTSUITE whereas qam uses QA_TESTSET only. Checked with https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module and found no QAM job running execute_test_run. https://openqa.suse.de/admin/test_suites shows that there are mainly QAM jobs using QA_TESTSET however some others as well. QA_TESTSUITE is not used for anything with a prefix like "qam". To me it looks like both are doing more or less the equivalent except that based on QA_TESTSET shows the output of individual test modules and also there is no module "execute_test_run" that would fail and is hard to label.

Can I just define QA_TESTSET instead of QA_TESTSUITE?

$ openqa_clone_job_osd --skip-chained-deps 2488712 QA_TESTSUITE= QA_TESTSET=userspace_apparmor _GROUP=0 TEST=okurz_poo44138_qa_userspace_apparmor_testset

Created job #2492870: sle-15-SP1-Installer-DVD-x86_64-Build175.4-qa_userspace_apparmor@64bit -> https://openqa.suse.de/t2492870

#7 Updated by okurz over 1 year ago

There seems to be no "python" in https://openqa.suse.de/tests/2492870#step/userspace_apparmor/48 . That was unexpected :D

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6878 to fix first before we could switch the scenarios to the different run method.

For direct comparison:

#8 Updated by okurz over 1 year ago

  • Blocks action #44828: [sle][functional][u][userspace] test fails in execute_test_run (test_verify) - need to update test code and report new test result added

#10 Updated by okurz over 1 year ago

#11 Updated by okurz over 1 year ago

  • Blocks action #44195: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed added

#12 Updated by okurz over 1 year ago

  • Blocks action #44882: [functional][sle][userspace][u]test fails in execute_test_run - almost all php tests failed added

#13 Updated by okurz over 1 year ago

both mentioned PRs merged. Maintenance tests still look very green, e.g. https://openqa.suse.de/tests/overview?distri=sle&version=12-SP3&build=20190301-1&groupid=108 . Further examples: mau-qa_acceptance_fs_stress and mau-qa_userspace_bind

Changing a limited set of test suites, e.g. "qa_userspace_apparmor":

-VIRTIO_CONSOLE=0
-MAX_JOB_TIME=14400
-QA_HEAD_REPO=http://dist.nue.suse.com/ibs/QA:/Head/SLE-%VERSION%
-QA_TESTSUITE=apparmor
+QA_TESTSET=userspace_apparmor

Also adopted "qa_userspace_apparmor-profiles".

Let's wait for a new build.

#14 Updated by okurz over 1 year ago

  • Related to action #48653: [functional][u] test fails in kernel_multipath and mau-qa_kernel_kexec on SLE-12 SP3 and newer added

#15 Updated by okurz over 1 year ago

One side-effect is #48653.

#16 Updated by okurz over 1 year ago

for i in 2513547 2513388 2513180 2513222 2513567 ; do openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i QA_HEAD_REPO=http://dist.nue.suse.com/ibs/QA:/Head/SLE-15-SP1 ; done

Created job #2518055: sle-15-SP1-Installer-DVD-x86_64-Build181.3-qa_userspace_apparmor@64bit -> http://openqa.suse.de/t2518055 -> now failing as expected in the same bug as in before
Created job #2518056: sle-15-SP1-Installer-DVD-aarch64-Build181.3-qa_userspace_apparmor-profiles@aarch64-virtio -> http://openqa.suse.de/t2518056
Created job #2518057: sle-15-SP1-Installer-DVD-ppc64le-Build181.3-qa_userspace_apparmor-profiles@ppc64le -> http://openqa.suse.de/t2518057
Created job #2518058: sle-15-SP1-Installer-DVD-s390x-Build181.3-qa_userspace_apparmor-profiles@s390x-kvm-sle12 -> http://openqa.suse.de/t2518058
Created job #2518059: sle-15-SP1-Installer-DVD-x86_64-Build181.3-qa_userspace_apparmor-profiles@64bit -> http://openqa.suse.de/t2518059 -> failed weirdly in an intermediate step, retrying: https://openqa.suse.de/tests/2518080 . Failed the same however I have seen the same error in https://openqa.suse.de/tests/2518055#step/userspace_apparmor/30 without any bad effect so the error must be something different. https://openqa.suse.de/tests/2518080/file/serial_terminal.txt shows

# /usr/share/qa/qaset/bin/junit_xml_gen.py -n 'regression' -d -o /tmp/junit.xml /var/log/qaset ; echo _dMLG-$?-
[/usr/share/qa/qaset/bin/junit_xml_gen.py]WARNING: No submission info for testsuite apparmor-profiles
[/usr/share/qa/qaset/bin/junit_xml_gen.py]WARNING: No log dir inside submission log of testsuite apparmor-profiles
[/usr/share/qa/qaset/bin/junit_xml_gen.py]DEBUG: Parsing testsuite iostat: iostat
Traceback (most recent call last):
  File "/usr/share/qa/qaset/bin/junit_xml_gen.py", line 512, in <module>
    dc.collect_log()
  File "/usr/share/qa/qaset/bin/junit_xml_gen.py", line 426, in collect_log
    shutil.rmtree(ts_dir)
  File "/usr/lib64/python2.7/shutil.py", line 253, in rmtree
    onerror(os.listdir, path, sys.exc_info())
  File "/usr/lib64/python2.7/shutil.py", line 251, in rmtree
    names = os.listdir(path)
OSError: [Errno 20] Not a directory: '/tmp/junit_generator-66651/iostat'

so for now I am changing "apparmor-profiles" back to the original.

I suggest the following next step:

  • Test all scenarios in "userspace" with QA_TESTSET=userspace-<name> and VIRTIO_CONSOLE=1 instead of QA_TESTSUITE=<name>
  • understand why some do not work, e.g. as above "apparmor-profiles"
  • @waitfor Ask others if QA_TESTSUITE has any benefit over QA_TESTSET -> propose deletion and ask in PR -> @waitfor https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6955
  • @waitfor ^ Change all/more scenarios to QA_TESTSET instead

#17 Updated by okurz over 1 year ago

  • Blocks action #46490: [functional][u][sporadic] test fails in execute_test_run: char gets double-typed added

#18 Updated by okurz over 1 year ago

Trying to clone all scenarios from latest 15sp1 build with testset:

for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output  '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace-${name}" TEST=qa_userspace_${name}-testset-poo44138 _GROUP="Test Development: SLE 15" ;done

-> https://openqa.suse.de/tests/overview?distri=sle&build=189.1&version=15-SP1&groupid=96

Hm, they all failed. Probably because I used "userspace-${name}" instead of "userspace_${name}". Also the results are actually hard to see in the development job group, trying again in it's own (implicit) build:

for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output  '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace_${name}" TEST=qa_userspace_${name}-testset-poo44138 _GROUP=0 ;done

-> Created job #2537657: sle-15-SP1-Installer-DVD-aarch64-Build189.1-qa_userspace_apache2_mod_perl@aarch64-virtio -> http://openqa.suse.de/t2537657

hm, that was also a bad idea as this is now the same build but without any group -> https://openqa.suse.de/tests/overview?version=15-SP1&build=189.1&distri=sle

As the tests do not take that long I deleted some still scheduled ones:

for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 state=scheduled arch=s390x | jq --raw-output --join-output  '.jobs[] | (.id," ")') ; do openqa-client --host openqa.suse.de jobs/$i delete ; done

and I retriggered with a custom build:

for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output  '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace_${name}" TEST=qa_userspace_${name}-testset-poo44138 BUILD=189.1@poo44138 _GROUP=0 ;done

Created job #2537763: sle-15-SP1-Installer-DVD-aarch64-Build189.1-qa_userspace_apache2_mod_perl@aarch64-virtio -> http://openqa.suse.de/t2537763 -> https://openqa.suse.de/tests/overview?version=15-SP1&build=189.1%40poo44138&distri=sle

The following testsuites seem to be problematic:

  • apache2-mod_perl -> #49106
  • apparmor-profiles (already mentioned above) -> #49106
  • sharutils (failed on s390x) -> #49112
  • php (should have been php7) -> #49115

the following fail but should be fine:

so I changed all the testsuites except the "problematic" ones as mentioned above to use "TESTSET" instead.

#19 Updated by okurz over 1 year ago

  • Copied to action #49118: [functional][u][userspace] Investigate if QA_TESTSUITE has any benefits over QA_TESTSET added

#20 Updated by okurz over 1 year ago

The only part left for this commit is I should await a next official SLE15SP1 build with the according test results in the userspace job group.

#21 Updated by okurz over 1 year ago

  • Status changed from Feedback to Resolved

#22 Updated by szarate 12 months ago

  • Blocks deleted (action #44195: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed)

#23 Updated by szarate 12 months ago

  • Blocked by action #44195: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed added

#24 Updated by szarate 12 months ago

  • Blocked by deleted (action #44195: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed)

#25 Updated by szarate 12 months ago

  • Precedes action #44195: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed added

Also available in: Atom PDF