https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842018-07-18T09:03:40ZopenSUSE Project Management ToolopenQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1362472018-07-18T09:03:40Zmkittlermarius.kittler@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/136247/diff?detail_id=137513">diff</a>)</li></ul> openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1368142018-07-19T15:18:48Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul><p>PRs:</p>
<ul>
<li>backend: <a href="https://github.com/os-autoinst/os-autoinst/pull/993" class="external">https://github.com/os-autoinst/os-autoinst/pull/993</a></li>
<li>UI: <a href="https://github.com/os-autoinst/openQA/pull/1727" class="external">https://github.com/os-autoinst/openQA/pull/1727</a></li>
</ul>
<hr>
<p>Seems to work roughly but still needs further testing and likely bug fixing. Extending the test-suite to cover this will be the challenge.</p>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1372612018-07-20T14:55:07Zmkittlermarius.kittler@suse.com
<ul></ul><p>Problems I've noticed so far:</p>
<ol>
<li>I tried to make it upload the screenshots for the current module inside <code>Worker::Jobs::upload_status</code>. This is required if one wants to open the needle editor to create a new needle when the assertion failed. However, my adjustments to <code>upload_status</code> don't work very well because during the pause at a certain module now two modules appear as running at the same time. I'll have to find out how to adjust <code>upload_status</code> to do the right thing.<br></li>
<li>So far os-autoinst pauses when a timeout occurs for any reason. However, it should only pause for <code>assert_screen</code>s and maybe (optionally) for <code>check_screen</code>s. There are also other functions like <code>send_key_until_needlematch</code> which call <code>check_screen</code> and <code>assert_screen</code> internally. Not sure whether those should be ignored, too.</li>
</ol>
<p>Log messages from worker and web UI which are related to 1.:</p>
<pre><code>can't open /tmp/wUm1C2CH2e/t/full-stack.d/openqa/pool/1/testresults/result-shutdown.json: No such file or directory at /opt/testing_area/openqa/script/../lib/OpenQA/Worker/Jobs.pm line 828.
Odd number of elements in anonymous hash at /opt/testing_area/openqa/script/../lib/OpenQA/Worker/Jobs.pm line 704.
DBIx::Class::Storage::DBI::select_single(): Query returned more than one row. SQL that returns multiple rows is DEPRECATED for ->find and ->single at /hdd/openqa-devel/repos/openQA/script/../lib/OpenQA/WebAPI/Controller/Running.pm line 54
</code></pre>
<p>(also visible in Travis log of WIP PR)</p>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1376872018-07-23T14:25:44Zmkittlermarius.kittler@suse.com
<ul></ul><p>Current state:</p>
<ul>
<li>See PRs mentioned in previous comment</li>
<li>What works:
<ul>
<li>setting/unsetting whether to pause on <code>assert_screen</code> failure via UI</li>
<li>pausing on <code>assert_screen</code> failure if enabled</li>
<li>failing as usual if pausing on <code>assert_screen</code> is not enabled</li>
<li>setting/unsetting whether to pause on <code>assert_screen</code> failure when already paused: test behaves as expected on resume</li>
<li>opening needle editor when paused and create a new needle</li>
<li>resume with new needle and successfully assert it</li>
<li>the 2nd point mentioned in the previous comment should be fixed now - it will now only pause on <code>assert_screen</code>s but not on <code>check_screen</code>s</li>
<li>the needle editor suggest to return to live view and not to restart the job if a development session for the job is present</li>
</ul></li>
<li>What doesn't work:
<ul>
<li>The 1st point mentioned in the last comment is still TODO. This problem is also causing the tests to fail.</li>
<li>At some point the browser window seems to frees. Maybe this only happens when the JavaScript debugger is used.</li>
<li>Failed tags are automatically added in needle editor (required?).</li>
</ul></li>
<li>Tests I've done far:
<ul>
<li>Add UI-only tests to <code>t/ui/25-developer-mode.t</code></li>
<li>Add unit tests for <code>assert_screen</code> and <code>check_screen</code> to <code>t/03-testapi.t</code></li>
<li>Otherwise only manual testing with 'opensuse-Staging:B-Staging-DVD-x86_64-Build1190.2-kde@64bit'
<ul>
<li>I adjusted it to fail very soon: <a href="https://github.com/Martchus/os-autoinst-distri-opensuse/commit/let_bootloader_fail">https://github.com/Martchus/os-autoinst-distri-opensuse/commit/let_bootloader_fail</a></li>
<li>Needles created with needle editor when paused: <a href="https://github.com/Martchus/os-autoinst-needles-opensuse/commit/let_bootloader_fail">https://github.com/Martchus/os-autoinst-needles-opensuse/commit/let_bootloader_fail</a></li>
</ul></li>
<li>So <code>t/33-developer-mode.t</code> still needs to be extended.
<ul>
<li>To get started, I would just rename some needle (eg. <code>boot-on_prompt.json</code>) so it can not be found anymore. If the test pauses, I would rename it and check whether the test continues on resume. It is likely the easiest to just use the devel console for this (instead of dealing with the UI elements). DONE</li>
<li>Then I'd continue with further things like opening the needle editor when paused.</li>
</ul></li>
</ul></li>
</ul>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1393642018-07-31T15:54:01Zmkittlermarius.kittler@suse.com
<ul></ul><p>Making the needle editor work is tricky. It basically works now but when adding a test for it (<a href="https://github.com/Martchus/openQA/commit/pause_on_assert_screen_timeout_upload_event" class="external">https://github.com/Martchus/openQA/commit/pause_on_assert_screen_timeout_upload_event</a>) I just get 404. When trying it again with a short delay it works. So the reason is that the required results haven't been uploaded quickly enough. During manual testing I was always slow enough not to run into this issue.</p>
<p>I remember that we had a similar problem with the old interactive mode (screenshot was sometimes not available when opening the needle editor).</p>
<p>To fix this we likely have to pass the progress of the result upload to the client. Then it can wait till the upload has been completed before offering to open the needle editor. This could be done using the web socket route.</p>
<hr>
<p>Can you give me any feedback about this so far? I'll deploy this on a staging instance so you can try yourself.</p>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1397932018-08-01T14:06:36Zmkittlermarius.kittler@suse.com
<ul></ul><p>I improved the 404 error pages shown when opening the needle editor for a running test. I think that's a cheap workaround for the problem mentioned in the previous comment and helpful anyways.</p>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1399552018-08-02T14:09:02Zcoolocoolo@suse.com
<ul></ul><p>IMO the client needs to be certain the uploads are in before opening a link to the needle editor.</p>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1415332018-08-09T08:23:26Zmkittlermarius.kittler@suse.com
<ul></ul><p>My ideas how to implement that:</p>
<ol>
<li>The worker could pass that info to the os-autoinst command server which could forward it to it the liveview handler which could forward it to the browser. That way the info is forwarded twice which is not so nice. Also, replaying the info for a newly joined client would be more effort compared to 2.</li>
<li>The worker could pass that info to the web UI using a new route <em>provided by the livehandler daemon</em>. The last part is important because only that way the info can be forwarded to the connected web browser clients via web sockets. The livehandler could also store the info in the database so it can be easily included to the status information every newly joining client receives at the beginning.</li>
</ol>
<p>I think 2. is the better option so I'm going to implement it if there are no objections :-)</p>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1418962018-08-10T09:54:54Zmkittlermarius.kittler@suse.com
<ul></ul><p>PR for implementing 2.: <a href="https://github.com/os-autoinst/openQA/pull/1745" class="external">https://github.com/os-autoinst/openQA/pull/1745</a></p>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1492822018-09-14T07:26:38Zcoolocoolo@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-6 priority-high2 closed" href="/issues/41015">action #41015</a>: Don't use livehandler if no developer looks at it</i> added</li></ul> openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1492882018-09-14T07:26:44Zcoolocoolo@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>This is implemented - but there are stability issues with it, we track in <a href="https://progress.opensuse.org/issues/41015" class="external">https://progress.opensuse.org/issues/41015</a></p>
openQA Project - action #38510: Allow os-autoinst to pause on next assert_screen timeouthttps://progress.opensuse.org/issues/38510?journal_id=1493782018-09-14T12:17:11Zcoolocoolo@suse.com
<ul><li><strong>Target version</strong> changed from <i>Current Sprint</i> to <i>Done</i></li></ul>