https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842018-11-21T09:13:05ZopenSUSE Project Management ToolopenQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1671172018-11-21T09:13:05Zokurzokurz@suse.com
<ul><li><strong>Parent task</strong> set to <i>#34663</i></li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1738462018-12-13T15:27:34Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-1 priority-4 priority-default parent" href="/issues/44843">coordination #44843</a>: [qe-core][functional][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module)</i> added</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1897972019-02-17T22:19:26Zokurzokurz@suse.com
<ul><li><strong>Target version</strong> changed from <i>Milestone 22</i> to <i>Milestone 24</i></li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1921672019-02-24T06:47:14Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p><a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4132" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4132</a> with title "qa_automation: add new design for QA:HEAD tests" added the use of the test variable QA_TESTSUITE whereas qam uses QA_TESTSET only. Checked with <a href="https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module" class="external">https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module</a> and found no QAM job running execute_test_run. <a href="https://openqa.suse.de/admin/test_suites" class="external">https://openqa.suse.de/admin/test_suites</a> shows that there are <em>mainly</em> QAM jobs using QA_TESTSET however some others as well. QA_TESTSUITE is not used for anything with a prefix like "qam". To me it looks like both are doing more or less the equivalent except that based on QA_TESTSET shows the output of individual test modules and also there is no module "execute_test_run" that would fail and is hard to label.</p>
<p>Can I just define QA_TESTSET instead of QA_TESTSUITE?</p>
<pre><code>$ openqa_clone_job_osd --skip-chained-deps 2488712 QA_TESTSUITE= QA_TESTSET=userspace_apparmor _GROUP=0 TEST=okurz_poo44138_qa_userspace_apparmor_testset
</code></pre>
<p>Created job #2492870: sle-15-SP1-Installer-DVD-x86_64-Build175.4-qa_userspace_apparmor@64bit -> <a href="https://openqa.suse.de/t2492870" class="external">https://openqa.suse.de/t2492870</a></p>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1921792019-02-24T09:40:11Zokurzokurz@suse.com
<ul></ul><p>There seems to be no "python" in <a href="https://openqa.suse.de/tests/2492870#step/userspace_apparmor/48" class="external">https://openqa.suse.de/tests/2492870#step/userspace_apparmor/48</a> . That was unexpected :D</p>
<p><a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6878" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6878</a> to fix first before we could switch the scenarios to the different run method.</p>
<p>For direct comparison:</p>
<ul>
<li>SLE15SP1 userspace regression test on production: <a href="https://openqa.suse.de/tests/2488712#" class="external">https://openqa.suse.de/tests/2488712#</a></li>
<li>New approach based on QA_TESTSET including above PR: <a href="https://openqa.suse.de/tests/2492918" class="external">https://openqa.suse.de/tests/2492918</a></li>
<li>QAM equivalent to above with QA_TESTSET: <a href="https://openqa.suse.de/tests/2492722" class="external">https://openqa.suse.de/tests/2492722</a></li>
</ul>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1921822019-02-24T09:41:09Zokurzokurz@suse.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/44828">action #44828</a>: [sle][functional][u][userspace] test fails in execute_test_run (test_verify) - need to update test code and report new test result</i> added</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1922002019-02-24T10:27:12Zokurzokurz@suse.com
<ul></ul><p><a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6879" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6879</a> is a more experimental PR, going further</p>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1932262019-02-26T14:16:31Zokurzokurz@suse.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/44195">action #44195</a>: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed</i> added</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1932322019-02-26T14:16:39Zokurzokurz@suse.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/44882">action #44882</a>: [functional][sle][userspace][u]test fails in execute_test_run - almost all php tests failed</i> added</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1943992019-03-01T13:23:19Zokurzokurz@suse.com
<ul></ul><p>both mentioned PRs merged. Maintenance tests still look very green, e.g. <a href="https://openqa.suse.de/tests/overview?distri=sle&version=12-SP3&build=20190301-1&groupid=108" class="external">https://openqa.suse.de/tests/overview?distri=sle&version=12-SP3&build=20190301-1&groupid=108</a> . Further examples: <a href="https://openqa.suse.de/tests/2507824#" class="external">mau-qa_acceptance_fs_stress</a> and <a href="https://openqa.suse.de/tests/2507796#" class="external">mau-qa_userspace_bind</a></p>
<p>Changing a limited set of test suites, e.g. "qa_userspace_apparmor":</p>
<pre><code>-VIRTIO_CONSOLE=0
-MAX_JOB_TIME=14400
-QA_HEAD_REPO=http://dist.nue.suse.com/ibs/QA:/Head/SLE-%VERSION%
-QA_TESTSUITE=apparmor
+QA_TESTSET=userspace_apparmor
</code></pre>
<p>Also adopted "qa_userspace_apparmor-profiles".</p>
<p>Let's wait for a new build.</p>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1950412019-03-04T16:20:12Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/48653">action #48653</a>: [functional][u] test fails in kernel_multipath and mau-qa_kernel_kexec on SLE-12 SP3 and newer</i> added</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1950472019-03-04T16:20:31Zokurzokurz@suse.com
<ul></ul><p>One side-effect is <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: [functional][u] test fails in kernel_multipath and mau-qa_kernel_kexec on SLE-12 SP3 and newer (Resolved)" href="https://progress.opensuse.org/issues/48653">#48653</a>.</p>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1950562019-03-04T16:28:54Zokurzokurz@suse.com
<ul></ul><ul>
<li>handle result of <a href="https://openqa.suse.de/tests/2513547#step/userspace_apparmor/15">https://openqa.suse.de/tests/2513547#step/userspace_apparmor/15</a> -> "# Test died: No QA_HEAD_REPO specified! at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/qa_automation/qa_run.pm line 84." -> I guess I still need <code>QA_HEAD_REPO</code> (for now). Readded <code>QA_HEAD_REPO=http://dist.nue.suse.com/ibs/QA:/Head/SLE-%VERSION%</code> to both testsuites changed in <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to re... (Resolved)" href="https://progress.opensuse.org/issues/44138#note-13">#44138#note-13</a></li>
</ul>
<pre><code>for i in 2513547 2513388 2513180 2513222 2513567 ; do openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i QA_HEAD_REPO=http://dist.nue.suse.com/ibs/QA:/Head/SLE-15-SP1 ; done
</code></pre>
<p>Created job #2518055: sle-15-SP1-Installer-DVD-x86_64-Build181.3-qa_userspace_apparmor@64bit -> <a href="http://openqa.suse.de/t2518055">http://openqa.suse.de/t2518055</a> -> now failing as expected in the same bug as in before<br>
Created job #2518056: sle-15-SP1-Installer-DVD-aarch64-Build181.3-qa_userspace_apparmor-profiles@aarch64-virtio -> <a href="http://openqa.suse.de/t2518056">http://openqa.suse.de/t2518056</a><br>
Created job #2518057: sle-15-SP1-Installer-DVD-ppc64le-Build181.3-qa_userspace_apparmor-profiles@ppc64le -> <a href="http://openqa.suse.de/t2518057">http://openqa.suse.de/t2518057</a><br>
Created job #2518058: sle-15-SP1-Installer-DVD-s390x-Build181.3-qa_userspace_apparmor-profiles@s390x-kvm-sle12 -> <a href="http://openqa.suse.de/t2518058">http://openqa.suse.de/t2518058</a><br>
Created job #2518059: sle-15-SP1-Installer-DVD-x86_64-Build181.3-qa_userspace_apparmor-profiles@64bit -> <a href="http://openqa.suse.de/t2518059">http://openqa.suse.de/t2518059</a> -> failed weirdly in an intermediate step, retrying: <a href="https://openqa.suse.de/tests/2518080">https://openqa.suse.de/tests/2518080</a> . Failed the same however I have seen the same error in <a href="https://openqa.suse.de/tests/2518055#step/userspace_apparmor/30">https://openqa.suse.de/tests/2518055#step/userspace_apparmor/30</a> without any bad effect so the error must be something different. <a href="https://openqa.suse.de/tests/2518080/file/serial_terminal.txt">https://openqa.suse.de/tests/2518080/file/serial_terminal.txt</a> shows</p>
<pre><code># /usr/share/qa/qaset/bin/junit_xml_gen.py -n 'regression' -d -o /tmp/junit.xml /var/log/qaset ; echo _dMLG-$?-
[/usr/share/qa/qaset/bin/junit_xml_gen.py]WARNING: No submission info for testsuite apparmor-profiles
[/usr/share/qa/qaset/bin/junit_xml_gen.py]WARNING: No log dir inside submission log of testsuite apparmor-profiles
[/usr/share/qa/qaset/bin/junit_xml_gen.py]DEBUG: Parsing testsuite iostat: iostat
Traceback (most recent call last):
File "/usr/share/qa/qaset/bin/junit_xml_gen.py", line 512, in <module>
dc.collect_log()
File "/usr/share/qa/qaset/bin/junit_xml_gen.py", line 426, in collect_log
shutil.rmtree(ts_dir)
File "/usr/lib64/python2.7/shutil.py", line 253, in rmtree
onerror(os.listdir, path, sys.exc_info())
File "/usr/lib64/python2.7/shutil.py", line 251, in rmtree
names = os.listdir(path)
OSError: [Errno 20] Not a directory: '/tmp/junit_generator-66651/iostat'
</code></pre>
<p>so for now I am changing "apparmor-profiles" back to the original.</p>
<p>I suggest the following next step:</p>
<ul>
<li>Test all scenarios in "userspace" with <code>QA_TESTSET=userspace-<name></code> and <code>VIRTIO_CONSOLE=1</code> instead of <code>QA_TESTSUITE=<name></code></li>
<li>understand why some do not work, e.g. as above "apparmor-profiles"</li>
<li>@waitfor Ask others if QA_TESTSUITE has any benefit over QA_TESTSET -> propose deletion and ask in PR -> @waitfor <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6955">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6955</a></li>
<li>@waitfor <sup>^</sup> Change all/more scenarios to QA_TESTSET instead</li>
</ul>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1952002019-03-05T06:50:04Zokurzokurz@suse.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/46490">action #46490</a>: [functional][u][sporadic] test fails in execute_test_run: char gets double-typed</i> added</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1984732019-03-12T19:57:28Zokurzokurz@suse.com
<ul></ul><p>Trying to clone all scenarios from latest 15sp1 build with testset:</p>
<pre><code>for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace-${name}" TEST=qa_userspace_${name}-testset-poo44138 _GROUP="Test Development: SLE 15" ;done
</code></pre>
<p>-> <a href="https://openqa.suse.de/tests/overview?distri=sle&build=189.1&version=15-SP1&groupid=96">https://openqa.suse.de/tests/overview?distri=sle&build=189.1&version=15-SP1&groupid=96</a></p>
<p>Hm, they all failed. Probably because I used "userspace-${name}" instead of "userspace_${name}". Also the results are actually hard to see in the development job group, trying again in it's own (implicit) build:</p>
<pre><code>for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace_${name}" TEST=qa_userspace_${name}-testset-poo44138 _GROUP=0 ;done
</code></pre>
<p>-> Created job #2537657: sle-15-SP1-Installer-DVD-aarch64-Build189.1-qa_userspace_apache2_mod_perl@aarch64-virtio -> <a href="http://openqa.suse.de/t2537657">http://openqa.suse.de/t2537657</a></p>
<p>hm, that was also a bad idea as this is now the same build but without any group -> <a href="https://openqa.suse.de/tests/overview?version=15-SP1&build=189.1&distri=sle">https://openqa.suse.de/tests/overview?version=15-SP1&build=189.1&distri=sle</a></p>
<p>As the tests do not take that long I deleted some still scheduled ones:</p>
<pre><code>for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 state=scheduled arch=s390x | jq --raw-output --join-output '.jobs[] | (.id," ")') ; do openqa-client --host openqa.suse.de jobs/$i delete ; done
</code></pre>
<p>and I retriggered with a custom build:</p>
<pre><code>for i in $(openqa-client --host openqa.suse.de --json-output jobs build=189.1 version=15-SP1 groupid=112| jq --raw-output --join-output '.jobs[]| (.id," ")') ; do name=$(openqa-client --host openqa.suse.de --json-output jobs/$i | jq -r '.job.settings.QA_TESTSUITE'); openqa-clone-job --skip-chained-deps --within-instance openqa.suse.de $i VIRTIO_CONSOLE= MAX_JOB_TIME= QA_TESTSUITE= QA_TESTSET="userspace_${name}" TEST=qa_userspace_${name}-testset-poo44138 BUILD=189.1@poo44138 _GROUP=0 ;done
</code></pre>
<p>Created job #2537763: sle-15-SP1-Installer-DVD-aarch64-Build189.1-qa_userspace_apache2_mod_perl@aarch64-virtio -> <a href="http://openqa.suse.de/t2537763">http://openqa.suse.de/t2537763</a> -> <a href="https://openqa.suse.de/tests/overview?version=15-SP1&build=189.1%40poo44138&distri=sle">https://openqa.suse.de/tests/overview?version=15-SP1&build=189.1%40poo44138&distri=sle</a></p>
<p>The following testsuites seem to be problematic:</p>
<ul>
<li>apache2-mod_perl -> <a class="issue tracker-4 status-3 priority-3 priority-lowest closed" title="action: [functional][u][userspace] Convert "qa_userspace_apache2_mod_perl" to "console/apache2_mod_perl" (Resolved)" href="https://progress.opensuse.org/issues/49106">#49106</a></li>
<li>apparmor-profiles (already mentioned above) -> <a class="issue tracker-4 status-3 priority-3 priority-lowest closed" title="action: [functional][u][userspace] Convert "qa_userspace_apache2_mod_perl" to "console/apache2_mod_perl" (Resolved)" href="https://progress.opensuse.org/issues/49106">#49106</a></li>
<li>sharutils (failed on s390x) -> <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: [functional][u][userspace] Convert "qa_userspace_sharutils" to "console/sharutils" (Resolved)" href="https://progress.opensuse.org/issues/49112">#49112</a></li>
<li>php (should have been php7) -> <a class="issue tracker-4 status-3 priority-3 priority-lowest closed" title="action: [functional][u][userspace] Convert "qa_userspace_php7" to "console/php7" (Resolved)" href="https://progress.opensuse.org/issues/49115">#49115</a></li>
</ul>
<p>the following fail but should be fine:</p>
<ul>
<li>nfs_v4 (was scheduled as "null", should be checked specifically)</li>
<li>openssh failed the same as in before <a href="https://openqa.suse.de/tests/2533808">https://openqa.suse.de/tests/2533808</a></li>
<li>systemd fails the same as in before <a href="https://openqa.suse.de/tests/2536713">https://openqa.suse.de/tests/2536713</a></li>
</ul>
<p>so I changed all the testsuites except the "problematic" ones as mentioned above to use "TESTSET" instead.</p>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1984762019-03-12T19:59:09Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-6 priority-3 priority-lowest closed" href="/issues/49118">action #49118</a>: [functional][u][userspace] Investigate if QA_TESTSUITE has any benefits over QA_TESTSET</i> added</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1984852019-03-12T20:00:11Zokurzokurz@suse.com
<ul></ul><p>The only part left for this commit is I should await a next official SLE15SP1 build with the according test results in the userspace job group.</p>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=1999402019-03-15T14:24:00Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p><a href="https://openqa.suse.de/tests/overview?distri=sle&version=15-SP1&groupid=112" class="external">https://openqa.suse.de/tests/overview?distri=sle&version=15-SP1&groupid=112</a> looks like it should :)</p>
openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=2384332019-08-26T11:56:08Zszarate
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/44195">action #44195</a>: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed</i>)</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=2384482019-08-26T11:56:35Zszarate
<ul><li><strong>Blocked by</strong> <i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/44195">action #44195</a>: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed</i> added</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=2384542019-08-26T11:56:45Zszarate
<ul><li><strong>Blocked by</strong> deleted (<i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/44195">action #44195</a>: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed</i>)</li></ul> openQA Tests - action #44138: [functional][u][userspace] tests in QAM which seem to be the equivalent look so much easier to review, crosscheckhttps://progress.opensuse.org/issues/44138?journal_id=2384572019-08-26T11:57:05Zszarate
<ul><li><strong>Precedes</strong> <i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/44195">action #44195</a>: [sle][u] test fails in execute_test_run - check-named and check-systemd-machined failed</i> added</li></ul>