https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842019-03-05T11:57:00ZopenSUSE Project Management ToolopenQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=1955392019-03-05T11:57:00Zokurzokurz@suse.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-4 status-6 priority-4 priority-default closed" href="/issues/48383">action #48383</a>: [opensuse] test fails in first_boot - does not login in GDM</i> added</li></ul> openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=1962382019-03-06T10:28:50ZSLindoMansillaslindomansilla@suse.com
<ul><li><strong>Category</strong> set to <i>Spike/Research</i></li></ul><p>As a result of backlog triaging (see <a href="https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging" class="external">https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging</a> for more information).</p>
<p>Please, feel free to adjust the category if you think different.</p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2068102019-04-16T15:48:38Zggardet_armguillaume.gardet@arm.com
<ul></ul><p>Another occurrence: <a href="https://openqa.opensuse.org/tests/909526#step/first_boot/19" class="external">https://openqa.opensuse.org/tests/909526#step/first_boot/19</a></p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2210932019-06-18T07:15:21Zggardet_armguillaume.gardet@arm.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-4 status-1 priority-3 priority-lowest" href="/issues/53192">action #53192</a>: [opensuse] bootbasetest - Undefined subroutine &bootbasetest::export_logs</i> added</li></ul> openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2211052019-06-18T07:19:54Zggardet_armguillaume.gardet@arm.com
<ul></ul><p>Another occurrence: <a href="https://openqa.opensuse.org/tests/961187">https://openqa.opensuse.org/tests/961187</a></p>
<p><code><br>
[2019-06-16T15:56:38.769 CEST] [debug] Migrating the machine (Current VM state is running).<br>
[2019-06-16T15:56:38.793 CEST] [debug] EVENT {"event":"STOP","timestamp":{"microseconds":792862,"seconds":1560693398}}<br>
[2019-06-16T15:56:38.815 CEST] [debug] EVENT {"data":{"status":"setup"},"event":"MIGRATION","timestamp":{"microseconds":815366,"seconds":1560693398}}<br>
[2019-06-16T15:56:39.316 CEST] [debug] EVENT {"data":{"pass":1},"event":"MIGRATION_PASS","timestamp":{"microseconds":817443,"seconds":1560693398}}<br>
[2019-06-16T15:56:39.316 CEST] [debug] EVENT {"data":{"status":"active"},"event":"MIGRATION","timestamp":{"microseconds":817581,"seconds":1560693398}}<br>
[2019-06-16T15:56:39.317 CEST] [debug] Migrating total bytes: 2281992192<br>
[2019-06-16T15:56:39.317 CEST] [debug] Migrating remaining bytes: 1847336960<br>
[2019-06-16T15:56:39.818 CEST] [debug] Migrating total bytes: 2281992192<br>
[2019-06-16T15:56:39.818 CEST] [debug] Migrating remaining bytes: 1409036288<br>
[2019-06-16T15:56:40.320 CEST] [debug] Migrating total bytes: 2281992192<br>
[2019-06-16T15:56:40.320 CEST] [debug] Migrating remaining bytes: 953995264<br>
[2019-06-16T15:56:40.821 CEST] [debug] Migrating total bytes: 2281992192<br>
[2019-06-16T15:56:40.821 CEST] [debug] Migrating remaining bytes: 518422528<br>
[2019-06-16T15:56:41.323 CEST] [debug] Migrating total bytes: 2281992192<br>
[2019-06-16T15:56:41.323 CEST] [debug] Migrating remaining bytes: 166006784<br>
[2019-06-16T15:56:41.824 CEST] [debug] EVENT {"data":{"pass":2},"event":"MIGRATION_PASS","timestamp":{"microseconds":369398,"seconds":1560693401}}<br>
[2019-06-16T15:56:41.824 CEST] [debug] EVENT {"data":{"pass":3},"event":"MIGRATION_PASS","timestamp":{"microseconds":370524,"seconds":1560693401}}<br>
[2019-06-16T15:56:41.824 CEST] [debug] EVENT {"data":{"status":"completed"},"event":"MIGRATION","timestamp":{"microseconds":428071,"seconds":1560693401}}<br>
[2019-06-16T15:56:41.824 CEST] [debug] Migrating total bytes: 2281992192<br>
[2019-06-16T15:56:41.824 CEST] [debug] Migrating remaining bytes: 0<br>
[2019-06-16T15:56:41.830 CEST] [debug] Memory dump completed.<br>
[2019-06-16T15:56:42.309 CEST] [debug] EVENT {"event":"RESUME","timestamp":{"microseconds":309626,"seconds":1560693402}}<br>
ulogs/first_boot-vm-memory-dump: 416.3 MiB / 1,908.6 MiB = 0.218, 15 MiB/s, 2:08<br>
[2019-06-16T15:58:51.626 CEST] [debug] backend process exited: 0<br>
[2019-06-16T15:58:51.626 CEST] [debug] sysread failed: <br>
[2019-06-16T15:58:51.629 CEST] [debug] THERE IS NOTHING TO READ 15 4 3<br>
[2019-06-16T15:58:51.630 CEST] [debug] killing command server 9552 because test execution ended<br>
[2019-06-16T15:58:51.630 CEST] [debug] isotovideo: informing websocket clients before stopping command server: http://127.0.0.1:20113/gBJhrzNMsKyjjCkh/broadcast<br>
last frame<br>
[2019-06-16T15:58:51.689 CEST] [debug] Driver backend collected unknown process with pid 9572 and exit status: 0<br>
[2019-06-16T15:59:06.655 CEST] [debug] isotovideo: unable to inform websocket clients about stopping command server: Request timeout at /usr/bin/isotovideo line 172.<br>
[2019-06-16T15:59:07.660 CEST] [error] can_read received kill signal at /usr/lib/os-autoinst/myjsonrpc.pm line 91.<br>
</code></p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2213632019-06-18T11:51:50Zggardet_armguillaume.gardet@arm.com
<ul></ul><p>I suspect the problem is once dump file compression is done, the file is removed, but QEMU is still restoring the RAM from this file.<br>
So, I will test with <code>--keep</code> option for <code>xz</code>/<code>bzip2</code>.</p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2217892019-06-19T14:27:06Zggardet_armguillaume.gardet@arm.com
<ul></ul><p>Adding <code>--keep</code> option to xz/bzip2 does not help.<br>
Moving the <code>cont_vm</code> call later either. <br>
The problem is encountered once/after the <code>cont_vm</code> function is called. I do not know where exactly, but once <code>isotovideo</code> fails, qemu is still running. So, this is probably an internal <code>isotovideo</code> process which fails.</p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2218282019-06-19T16:12:09Zggardet_armguillaume.gardet@arm.com
<ul></ul><p>The read failure is from: <a href="https://github.com/os-autoinst/os-autoinst/blob/927f3d85e5440290d6aaa294b88c81a319abb99f/isotovideo#L398" class="external">https://github.com/os-autoinst/os-autoinst/blob/927f3d85e5440290d6aaa294b88c81a319abb99f/isotovideo#L398</a><br>
Not sure why there is nothing to read, though. Any idea?</p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2270782019-07-15T08:52:14Zggardet_armguillaume.gardet@arm.com
<ul></ul><p>I tried to freeze the VM before migration, but it did not help.<br>
I am running out of idea.</p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2351632019-08-14T12:47:33Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-6 priority-high2 closed" href="/issues/55505">action #55505</a>: lots of incompletes on o3, "qemu-system-x86_64: -vnc :91,share=force-shared: Failed to find an available port: Address already in use", maybe only openqaworker4:1?</i> added</li></ul> openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2351692019-08-14T12:47:36Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/42683">action #42683</a>: [functional][u] Make save_memory_dump work again and re-enable save_memory_dump call in tests/installation/first_boot and other boot modules</i> added</li></ul> openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2352142019-08-14T14:08:22Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>[aarch64] save_memory_dump make isotovideo to fail</i> to <i>[opensuse] save_memory_dump make isotovideo to fail</i></li><li><strong>Category</strong> changed from <i>Spike/Research</i> to <i>Bugs in existing tests</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>created <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8202" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8202</a> to prevent <code>save_memory_dump</code> causing more incompletes. Fails also on x86_64 e.g. <a href="https://openqa.opensuse.org/tests/1003323/file/autoinst-log.txt" class="external">https://openqa.opensuse.org/tests/1003323/file/autoinst-log.txt</a></p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2352802019-08-14T19:00:51Zokurzokurz@suse.com
<ul></ul><p><a href="https://openqa.opensuse.org/tests/1007345#" class="external">https://openqa.opensuse.org/tests/1007345#</a> is another example, merged the PR.</p>
<p>Monitoring:</p>
<ul>
<li><a href="https://openqa.opensuse.org/tests/1007890" class="external">https://openqa.opensuse.org/tests/1007890</a></li>
<li><a href="https://openqa.opensuse.org/tests/1007891" class="external">https://openqa.opensuse.org/tests/1007891</a></li>
<li><a href="https://openqa.opensuse.org/tests/1007892" class="external">https://openqa.opensuse.org/tests/1007892</a></li>
<li><a href="https://openqa.opensuse.org/tests/1007893" class="external">https://openqa.opensuse.org/tests/1007893</a></li>
<li><a href="https://openqa.opensuse.org/tests/1007894" class="external">https://openqa.opensuse.org/tests/1007894</a></li>
</ul>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2361772019-08-19T09:08:53Zcfconradcfamullaconrad@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed" href="/issues/55595">action #55595</a>: [cloud][pcm] debug memory dump - placeholder</i> added</li></ul> openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2361862019-08-19T09:17:55Zcfconradcfamullaconrad@suse.com
<ul></ul><p>Related change in os-autoinst: <a href="https://github.com/os-autoinst/os-autoinst/pull/1201" class="external">https://github.com/os-autoinst/os-autoinst/pull/1201</a></p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2403052019-09-02T12:08:50Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>(comment deleted)</p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2453002019-09-23T12:44:22Zszarate
<ul></ul><p>Memory dumps are working again: <a href="https://openqa.suse.de/tests/3393456#step/first_boot/14" class="external">https://openqa.suse.de/tests/3393456#step/first_boot/14</a> I guess Clemen's changes fixed the problem, so I re-enabled it.</p>
openQA Tests - action #48671: [opensuse] save_memory_dump make isotovideo to failhttps://progress.opensuse.org/issues/48671?journal_id=2454592019-09-23T20:46:07Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>Sure, but I would be a bit more careful, e.g. check on aarch64 and such. However, this ticket was about "save_memory_dump" makes tests fail which it didn't after I disabled it. That part is done :)</p>