https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842017-09-19T08:24:42ZopenSUSE Project Management ToolopenQA Tests - action #25404: [opensuse][o3]openQA's failure rate on typing commands increased againhttps://progress.opensuse.org/issues/25404?journal_id=647682017-09-19T08:24:42Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>openQA's failure rate on typing commands increased again</i> to <i>[opensuse][o3]openQA's failure rate on typing commands increased again</i></li><li><strong>Category</strong> changed from <i>Bugs in existing tests</i> to <i>Infrastructure</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li></ul><p>IRC log from <a href="irc://chat.freenode.net/opensuse-factory" class="external">#opensuse-factory</a>:</p>
<pre><code>[18 Sep 2017 17:30:20] <DimStar> https://openqa.opensuse.org/tests/487699#step/setup_zdup/20 - somebody trying to change the prompt again?
[18 Sep 2017 17:33:19] <DimStar> seems the code is fine - but openQA is very drunk when typing (test run 487697 typed it differently wrong)
[18 Sep 2017 17:34:36] <DimStar> sysrich: I don't think that update was a good idea; openQA can't type at all anymore!?! https://openqa.opensuse.org/tests/487691#step/user_settings/4 (password does not match on the confirm field)
[18 Sep 2017 17:35:26] <sysrich> DimStar, himm..thats weird, there is no major changes to the worker AFAIK..okurz?
[18 Sep 2017 17:35:44] <DimStar> hm.. strange; well, in worst case it IS a product bug; but good luck with that :)
[18 Sep 2017 17:37:01] <sysrich> DimStar, "at all" is an exaggeration - https://openqa.opensuse.org/tests/487685
[18 Sep 2017 17:37:29] <DimStar> sysrich: ok, it has a few cases it manages to type stuff right
[18 Sep 2017 17:38:10] <sysrich> DimStar, worker1 seems fine, it's worker4 that seems overloaded
[18 Sep 2017 17:38:16] <sysrich> by a factor of about 70% more load than worker1
[18 Sep 2017 17:39:00] <okurz> I don't recall any related changes to os-autoinst, maybe there was a qemu update?
[18 Sep 2017 17:39:13] <okurz> No, does not seem like it
[18 Sep 2017 17:39:34] <tacit> Yes there was a qemu updater.
[18 Sep 2017 17:39:49] <okurz> aha!
[18 Sep 2017 17:40:35] <sysrich> yeah we got the latest SLE/Leap qemu updates
[18 Sep 2017 17:40:49] <sysrich> okurz, what qemu do we have on osd?
[18 Sep 2017 17:41:18] <okurz> sysrich: qemu-2.6.2-31.3.3 from openSUSE-Leap-42.2-Update installed today
[…]
[18 Sep 2017 17:43:31] <okurz> the version is on o3. osd has 2.6.2-29.4
[18 Sep 2017 17:44:16] <okurz> well then, maybe downgrade one worker and hammer it over night?
</code></pre> openQA Tests - action #25404: [opensuse][o3]openQA's failure rate on typing commands increased againhttps://progress.opensuse.org/issues/25404?journal_id=647742017-09-19T08:57:38Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>SLindoMansilla</i></li></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/24132">@SLindoMansilla</a> who wants to help here. Please stay in close contact with people on <a href="irc://chat.freenode.net/opensuse-factory" class="external">#opensuse-factory</a>, especially DimStar and sysrich. nsinger, mgriessmeier, foursixnine might also be able help regarding administration.</p>
<p>For checking statistics it should be helpful to trigger more tests and check how many fail with new and old qemu version, e.g.</p>
<ul>
<li>trigger for i in {1..100}; do openqa_clone_job_o3 --skip-download --skip-chained-deps 488189 TEST=slindomansilla_check_mistyping_$i _GROUP="Development Tumbleweed" ; done</li>
<li>check how many fail with mistyping</li>
<li>downgrade qemu version on worker(s)</li>
<li>conduct check again</li>
<li>crosscheck statistics</li>
</ul>
openQA Tests - action #25404: [opensuse][o3]openQA's failure rate on typing commands increased againhttps://progress.opensuse.org/issues/25404?journal_id=647802017-09-19T09:03:41ZSLindoMansillaslindomansilla@suse.com
<ul></ul><p>Jobs with current qemu version scheduled: <a href="https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20170918&groupid=38" class="external">https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20170918&groupid=38</a></p>
openQA Tests - action #25404: [opensuse][o3]openQA's failure rate on typing commands increased againhttps://progress.opensuse.org/issues/25404?journal_id=648362017-09-19T15:01:27Zokurzokurz@suse.com
<ul></ul><p>sysrich also reported as -> <a href="https://bugzilla.suse.com/show_bug.cgi?id=1059369" class="external">https://bugzilla.suse.com/show_bug.cgi?id=1059369</a></p>
openQA Tests - action #25404: [opensuse][o3]openQA's failure rate on typing commands increased againhttps://progress.opensuse.org/issues/25404?journal_id=648562017-09-20T07:47:37ZSLindoMansillaslindomansilla@suse.com
<ul></ul><p>Currently no mistyping issues found after 100 jobs: <a href="https://openqa.opensuse.org/tests?match=slindomansilla_check_mistyping" class="external">https://openqa.opensuse.org/tests?match=slindomansilla_check_mistyping</a><br>
They were executed after the rollback.</p>
openQA Tests - action #25404: [opensuse][o3]openQA's failure rate on typing commands increased againhttps://progress.opensuse.org/issues/25404?journal_id=649002017-09-20T11:50:05ZSLindoMansillaslindomansilla@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Since the problem does not appear after the rollback, and after 100 successfully jobs, this ticket can be closed.</p>
openQA Tests - action #25404: [opensuse][o3]openQA's failure rate on typing commands increased againhttps://progress.opensuse.org/issues/25404?journal_id=662162017-09-29T19:43:41Zokurzokurz@suse.com
<ul></ul><p>tacit asked me if I can test a qemu update in <a href="http://download.opensuse.org/repositories/openSUSE:/Maintenance:/7326/openSUSE_Leap_42.2_Update/openSUSE:Maintenance:7326.repo" class="external">http://download.opensuse.org/repositories/openSUSE:/Maintenance:/7326/openSUSE_Leap_42.2_Update/openSUSE:Maintenance:7326.repo</a></p>
<p>so I added that repo to openqaworker4, replaced the worker class "qemu_x86_64" by "qemu_x86_64_okurz_incident_7326" and started tests with</p>
<pre><code>for i in {1..100}; do openqa_clone_job_o3 --skip-download --skip-chained-deps 494550 TEST=okurz_check_mistyping_poo25404_$i _GROUP="Development Tumbleweed" WORKER_CLASS=qemu_x86_64_okurz_incident_7326 ; done
</code></pre>
<p>will check results tomorrow.</p>
<p>EDIT: All jobs passed. Updated bug report <a href="https://bugzilla.suse.com/show_bug.cgi?id=1059369#c39" class="external">https://bugzilla.suse.com/show_bug.cgi?id=1059369#c39</a>. I removed the repo again and downgraded again to 2.6.2-29.4 to have a consistent state among all workers.</p>