https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842019-05-21T12:48:15ZopenSUSE Project Management ToolopenQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2136562019-05-21T12:48:15Zpvorelpetr.vorel@suse.com
<ul></ul><p>14:46 < nsinger> pvorel: the previous (working) runs where running with qemu-system-x86_64 -cpu qemu64 and now it runs with "-cpu host".</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2136952019-05-21T14:24:17ZSLindoMansillaslindomansilla@suse.com
<ul><li><strong>Category</strong> set to <i>Infrastructure</i></li></ul><p>As a result of backlog triaging (see <a href="https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging" class="external">https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging</a> for more information).</p>
<p>Please, feel free to adjust the category or the "[label]" if you think different.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2150392019-05-24T09:00:29Zpvorelpetr.vorel@suse.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2150752019-05-24T17:09:17Zpvorelpetr.vorel@suse.com
<ul></ul><p>Broken build <a href="https://openqa.opensuse.org/tests/940933/file/autoinst-log.txt" class="external">https://openqa.opensuse.org/tests/940933/file/autoinst-log.txt</a></p>
<pre><code>Use of uninitialized value in string eq at /usr/lib/os-autoinst/OpenQA/Qemu/DriveDevice.pm line 115.
Use of uninitialized value in string eq at /usr/lib/os-autoinst/OpenQA/Qemu/DriveDevice.pm line 115.
[2019-05-23T20:46:39.683 CEST] [debug] starting: /usr/bin/qemu-system-x86_64 -vga cirrus -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -soundhw ac97 -global isa-fdc.driveA= -m 1536 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot order=c,menu=on,splash-time=5000 -device usb-ehci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :104,share=force-shared -device virtio-serial -chardev socket,path=virtio_console,server,nowait,id=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-overlay0-file,filename=/var/lib/openqa/pool/14/raid/hd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0-overlay0,file=hd0-overlay0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0-overlay0,bootindex=0,serial=hd0 -blockdev driver=file,node-name=cd0-overlay0-file,filename=/var/lib/openqa/pool/14/raid/cd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=cd0-overlay0,file=cd0-overlay0-file,cache.no-flush=on -device scsi-cd,id=cd0-device,drive=cd0-overlay0,serial=cd0
Attempt 0 at /usr/lib/os-autoinst/osutils.pm line 130.
Attempt 1 at /usr/lib/os-autoinst/osutils.pm line 130.
</code></pre> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2189722019-06-07T12:11:56Zpvorelpetr.vorel@suse.com
<ul><li><strong>Subject</strong> changed from <i>[ltp][kernel][opensuse] All LTP tests are failing on openSUSE on x86_64</i> to <i>[ltp][kernel] All LTP tests are failing on SLE12 SP5 (osd) and openSUSE (o3) on x86_64</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/218972/diff?detail_id=216509">diff</a>)</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2189752019-06-07T12:12:37Zpvorelpetr.vorel@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/218975/diff?detail_id=216512">diff</a>)</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2189782019-06-07T12:18:14Zpvorelpetr.vorel@suse.com
<ul></ul><p>asmorodskyi found that QCOW2 image on osd was broken.<br>
Looking at it size is ok, just qemu-img check complains:</p>
<p>$ qemu-img check <a href="mailto:sle-12-SP5-x86_64-0187-Server-DVD@64bit-with-ltp.qcow2">sle-12-SP5-x86_64-0187-Server-DVD@64bit-with-ltp.qcow2</a><br>
qemu-img: This image format does not support checks</p>
<p>QEMU version which creates problems is <em>2.9.1(openSUSE Leap 42.3)</em>, but this was probably also on previous builds which didn't fail. On quasar.suse.cz, where I run install_ltp which produced a correct image is <em>2.11.2(SUSE Linux Enterprise 15)</em>. But really no clue what causes producing wrong qcow images.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2190112019-06-07T13:49:57Zpvorelpetr.vorel@suse.com
<ul></ul><p>On osd it helped to retrigger install_ltp.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2201692019-06-13T08:12:46Zpvorelpetr.vorel@suse.com
<ul></ul><p>Retriggering install_ltp helps on both osd and o3. Not sure what creates wrong image.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2214202019-06-18T13:49:14Zpcervinkapcervinka@suse.com
<ul></ul><p>Rechecked recent failed job <a href="https://openqa.suse.de/tests/2992911" class="external">https://openqa.suse.de/tests/2992911</a>, downloaded qcow2 image and tried to boot from in local virtual machine. Unfortunately it ended with "no bootable device error". It means that image is really corrupted and we need to find where it started. It can be incorrectly generated already on the worker or it can be corrupted during upload from worker.<br>
Both things can be difficult to troubleshoot.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2214232019-06-18T13:51:45Zpcervinkapcervinka@suse.com
<ul></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/28906">@pvorel</a> could you try to set variable QEMU_DISABLE_SNAPSHOTS in test install ltp test suite?</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2214502019-06-18T14:19:51Zrpalethorperichard.palethorpe@suse.com
<ul></ul><p>Just a thought, but if this is related to the problems I have seen before, then this is due to how the qcow images are stored and distributed. A solution might be to put them in a Ceph object store and allow it to handle as much of the distribution and caching as possible.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2218432019-06-19T20:12:28Zpvorelpetr.vorel@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-5 priority-4 priority-default closed" href="/issues/53294">action #53294</a>: [kernel][ltp] test fails in boot_ltp - incorrect kernel name provided</i> added</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2241622019-07-03T05:15:26Zokurzokurz@suse.com
<ul></ul><p>This is an autogenerated message for openQA integration by the openqa_review script:</p>
<p>This bug is still referenced in a failing openQA test: kernel-live-patching<br>
<a href="https://openqa.opensuse.org/tests/973499" class="external">https://openqa.opensuse.org/tests/973499</a></p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2249722019-07-04T14:23:57Zrpalethorperichard.palethorpe@suse.com
<ul></ul><p>Yeah, seems like the image on x86_64 o3 is always corrupt:</p>
<p>[2019-07-04T08:13:11.519 CEST] [debug] running /usr/bin/qemu-img info --output=json /var/lib/openqa/pool/2/<a href="mailto:opensuse-Tumbleweed-x86_64-20190703-DVD@64bit-with-ltp.qcow2">opensuse-Tumbleweed-x86_64-20190703-DVD@64bit-with-ltp.qcow2</a><br>
[2019-07-04T08:13:11.534 CEST] [debug] {<br>
"virtual-size": 2322137088,<br>
"filename": "/var/lib/openqa/pool/2/<a href="mailto:opensuse-Tumbleweed-x86_64-20190703-DVD@64bit-with-ltp.qcow2">opensuse-Tumbleweed-x86_64-20190703-DVD@64bit-with-ltp.qcow2</a>",<br>
"format": "raw",<br>
"actual-size": 2324410368,<br>
"dirty-flag": false<br>
}<br><br>
</p>
<p>Format should be qcow2.</p>
<p>[2019-07-04T09:19:53.726 UTC] [debug] {<br>
"virtual-size": 32212254720,<br>
"filename": "/var/lib/openqa/pool/6/<a href="mailto:opensuse-Tumbleweed-ppc64le-20190703-DVD@ppc64le-with-ltp.qcow2">opensuse-Tumbleweed-ppc64le-20190703-DVD@ppc64le-with-ltp.qcow2</a>",<br>
"cluster-size": 65536,<br>
"format": "qcow2",<br>
"actual-size": 2299990016,<br>
"format-specific": {<br>
"type": "qcow2",<br>
"data": {<br>
"compat": "1.1",<br>
"lazy-refcounts": false,<br>
"refcount-bits": 16,<br>
"corrupt": false<br>
}<br>
},<br>
"dirty-flag": false<br>
}</p>
<p>The publishing job doesn't take any snapshots.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2285452019-07-19T05:57:20Zokurzokurz@suse.com
<ul></ul><p>This is an autogenerated message for openQA integration by the openqa_review script:</p>
<p>This bug is still referenced in a failing openQA test: ltp_net_sctp<br>
<a href="https://openqa.suse.de/tests/3074428" class="external">https://openqa.suse.de/tests/3074428</a></p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2299192019-07-24T20:46:55Zpcervinkapcervinka@suse.com
<ul></ul><p>If we combine system installation with ltp installation, we will be able to save one job and image. Maybe we could create combined installation into one create_hdd_ltp which would do system installation with ltp at once. I see that there were some experiments already create_hdd_kotd_ltp, but is not used.</p>
<p>What do you think? </p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2300362019-07-25T09:41:32Zpvorelpetr.vorel@suse.com
<ul></ul><p>For VM based testing it worked ok, as we reused already installed image. So from these 3 steps (1) install OS 2) install LTP 3) run test) I'd prefer either join 2) + 3) and/or (only for IPMI) have all 3 steps in single test suite.</p>
<p>For IPMI I'm planning to use Michie's way (iPXE based installation), where he's going to detect which SLES version has been installed (so we avoid installing it if not needed).</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2335972019-08-09T05:23:55Zokurzokurz@suse.com
<ul></ul><p>This is an autogenerated message for openQA integration by the openqa_review script:</p>
<p>This bug is still referenced in a failing openQA test: kernel-live-patching<br>
<a href="https://openqa.opensuse.org/tests/1001688" class="external">https://openqa.opensuse.org/tests/1001688</a></p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2347402019-08-13T11:15:39Zpvorelpetr.vorel@suse.com
<ul></ul><p>Some time ago fixed in osd, but still broken on o3.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2396002019-08-30T06:05:22Zokurzokurz@suse.com
<ul></ul><p>This is an autogenerated message for openQA integration by the openqa_review script:</p>
<p>This bug is still referenced in a failing openQA test: ltp_net_ipv6_lib<br>
<a href="https://openqa.suse.de/tests/3308930" class="external">https://openqa.suse.de/tests/3308930</a></p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2412202019-09-04T09:06:36Zrpalethorperichard.palethorpe@suse.com
<ul><li><strong>Subject</strong> changed from <i>[ltp][kernel] All LTP tests are failing on SLE12 SP5 (osd) and openSUSE (o3) on x86_64</i> to <i>[ltp][kernel] All LTP tests are failing on SLE12 SP5 (osd) and openSUSE (o3) on [x86_64]</i></li></ul><p>Add arch tag for JDP.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2414332019-09-05T06:38:26Zpvorelpetr.vorel@suse.com
<ul><li><strong>Subject</strong> changed from <i>[ltp][kernel] All LTP tests are failing on SLE12 SP5 (osd) and openSUSE (o3) on [x86_64]</i> to <i>[ltp][kernel] All LTP tests are failing on openSUSE (o3) on [x86_64]</i></li></ul><p>Tests are failing just on Tumbleweed.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2527882019-10-25T11:15:51Zjlausuchjalausuch@suse.com
<ul><li><strong>Parent task</strong> set to <i>#58685</i></li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2527912019-10-25T11:17:01Zjlausuchjalausuch@suse.com
<ul><li><strong>Subject</strong> changed from <i>[ltp][kernel] All LTP tests are failing on openSUSE (o3) on [x86_64]</i> to <i>[kernel][ltp] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]</i></li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2610922019-12-02T03:36:46Zpvorelpetr.vorel@suse.com
<ul><li><strong>Assignee</strong> set to <i>pvorel</i></li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2610952019-12-02T03:37:12Zpvorelpetr.vorel@suse.com
<ul><li><strong>Target version</strong> set to <i>445</i></li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2614792019-12-03T11:28:58Zpvorelpetr.vorel@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2656982019-12-18T18:47:52Zpvorelpetr.vorel@suse.com
<ul><li><strong>Subject</strong> changed from <i>[kernel][ltp] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]</i> to <i>[kernel][ltp] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64][ppc64le]</i></li></ul><p>Hm, ppc64le got broken now as well:<br>
last good is build 20191206 (<a href="https://openqa.opensuse.org/tests/overview?version=Tumbleweed&build=20191206&groupid=32&distri=opensuse" class="external">https://openqa.opensuse.org/tests/overview?version=Tumbleweed&build=20191206&groupid=32&distri=opensuse</a>)<br>
first bad is build 20191216 (<a href="https://openqa.opensuse.org/tests/overview?distri=opensuse&groupid=32&build=20191216&version=Tumbleweed" class="external">https://openqa.opensuse.org/tests/overview?distri=opensuse&groupid=32&build=20191216&version=Tumbleweed</a>)</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2658122019-12-19T09:27:45Zpvorelpetr.vorel@suse.com
<ul></ul><p>There are other failures on ppc, maybe related <a href="https://openqa.suse.de/tests/3723347#next_previous" class="external">https://openqa.suse.de/tests/3723347#next_previous</a> (first fail: 4.12.14-146.1.ge31b461 <a href="https://openqa.suse.de/tests/3696777" class="external">https://openqa.suse.de/tests/3696777</a>)</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2658482019-12-19T10:02:17Zpvorelpetr.vorel@suse.com
<ul><li><strong>Subject</strong> changed from <i>[kernel][ltp] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64][ppc64le]</i> to <i>[kernel][ltp] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]</i></li></ul><p>ppc failures are caused by <a href="https://bugzilla.suse.com/show_bug.cgi?id=1159096" class="external">https://bugzilla.suse.com/show_bug.cgi?id=1159096</a> (on both osd and o3; checked by inspecting qcow2 image).</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2658902019-12-19T15:22:49Zpvorelpetr.vorel@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed parent" href="/issues/61203">coordination #61203</a>: [kernel][ltp][epic][grub] General solution for handling kernel parameters (debug_pagealloc=on)</i> added</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2659772019-12-20T06:17:04Zpvorelpetr.vorel@suse.com
<ul></ul><p>Hm, it's hard to debug this problem on o3 :(. I tried several times to restart a job and watch things on o3. Mostly I waited several hours and job got restarted while I was away. Today job got restarted, but I have no SSH access. According to okurz: ssh access to openqa.opensuse.org is down, see <a href="https://progress.opensuse.org/issues/61218" class="external">https://progress.opensuse.org/issues/61218</a>.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2694172020-01-09T09:33:15Zpvorelpetr.vorel@suse.com
<ul></ul><p>Still haven't found the root cause of the problem.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2707372020-01-13T15:44:19Zjlausuchjalausuch@suse.com
<ul></ul><p>can you reproduce it locally?</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2708632020-01-14T06:55:26Zpvorelpetr.vorel@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/270863/diff?detail_id=267017">diff</a>)</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2708662020-01-14T06:58:47Zpvorelpetr.vorel@suse.com
<ul></ul><p>jlausuch wrote:</p>
<blockquote>
<p>can you reproduce it locally?</p>
</blockquote>
<p>No, that's the hardest problem on this ticket (I thought I reported it, but I didn't), together with busyness of o3 (it's hard to reschedule the job).<br>
At least my PR which could help debugging a bit has been merged <a href="https://github.com/os-autoinst/os-autoinst/pull/1327" class="external">https://github.com/os-autoinst/os-autoinst/pull/1327</a>.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2708722020-01-14T07:06:14Zpvorelpetr.vorel@suse.com
<ul></ul><p>Hm, but looking at log (<a href="https://openqa.opensuse.org/tests/1142577/file/autoinst-log.txt" class="external">https://openqa.opensuse.org/tests/1142577/file/autoinst-log.txt</a>) of failing intel build (<a href="https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20200112&groupid=32" class="external">https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20200112&groupid=32</a>) published qcow2 image has qcow2 format, so the problem is somewhere else :(</p>
<pre><code>[2020-01-14T02:18:29.943 CET] [debug] running nice ionice qemu-img convert -c -O qcow2 /var/lib/openqa/pool/4/raid/hd0-overlay0 assets_public/opensuse-Tumbleweed-x86_64-20200112-DVD@64bit-with-ltp.qcow2
[2020-01-14T02:20:56.864 CET] [debug] running qemu-img info --output=json assets_public/opensuse-Tumbleweed-x86_64-20200112-DVD@64bit-with-ltp.qcow2
[2020-01-14T02:20:56.876 CET] [debug] {
"virtual-size": 32212254720,
"filename": "assets_public/opensuse-Tumbleweed-x86_64-20200112-DVD@64bit-with-ltp.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 1103364096,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false
}
},
"dirty-flag": false
}
</code></pre> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2710432020-01-14T10:43:20Zrpalethorperichard.palethorpe@suse.com
<ul></ul><p>To clarify: qemu-img shows that the qcow2 is valid after install-ltp completes, but when dependent test starts the image is invalid.</p>
<p>So the image is valid before it is uploaded as an asset, but is invalid by the time we download it.</p>
<p>using hexdump -n 1M -C :</p>
<pre><code>00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00040000
</code></pre>
<p>So the file is probably just all zeroes, but has the correct file length. For reference, it should look more like:</p>
<pre><code>hexdump -C -n 256K qa/runltp-support/ldisc-syzkaller.qcow2
00000000 51 46 49 fb 00 00 00 03 00 00 00 00 00 00 01 38 |QFI............8|
00000010 00 00 00 0b 00 00 00 10 00 00 00 0c 80 00 00 00 |................|
00000020 00 00 00 00 00 00 00 64 00 00 00 00 00 03 00 00 |.......d........|
00000030 00 00 00 00 00 01 00 00 00 00 00 01 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
...
</code></pre>
<p>There do not appear to be any errors during uploading. However it is possible that OpenQA is creating the file, but not writing to it correctly. Alternatively something could incorrectly copy the file after it has been uploaded as part of the asset caching.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2710492020-01-14T10:53:01Zrpalethorperichard.palethorpe@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-6 priority-5 priority-high3 closed" href="/issues/45836">action #45836</a>: [tools] qcow images mismatch in size </i> added</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2710522020-01-14T10:56:13Zpvorelpetr.vorel@suse.com
<ul></ul><p>rpalethorpe wrote:</p>
<blockquote>
<p>To clarify: qemu-img shows that the qcow2 is valid after install-ltp completes, but when dependent test starts the image is invalid.</p>
<p>So the image is valid before it is uploaded as an asset, but is invalid by the time we download it.</p>
</blockquote>
<p>Yep, I found that as well before, but just didn't believe it could be possible (so planned to investigate it more). mdoucha fount that both install_ltp+opensuse+DVD and install_ltp+opensuse+DVD-m32 used the same PUBLISH_HDD_1 and PUBLISH_PFLASH_VARS variables (thanks Martin!). While PUBLISH_PFLASH_VARS might not be a problem PUBLISH_HDD_1 certainly is. I restarted jobs with correct variables, let's see.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2710732020-01-14T11:42:42Zrpalethorperichard.palethorpe@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/34597">action #34597</a>: Race condition causing problems with the worker cache</i> added</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2710762020-01-14T11:44:56Zrpalethorperichard.palethorpe@suse.com
<ul><li><strong>Subject</strong> changed from <i>[kernel][ltp] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]</i> to <i>[openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]</i></li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2710852020-01-14T11:48:33Zrpalethorperichard.palethorpe@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-12 priority-3 priority-lowest" href="/issues/13646">action #13646</a>: Ensuring asset files integrity (was: "An error occurred during the installation" on images)</i> added</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2713792020-01-15T06:55:01Zpvorelpetr.vorel@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>Build 20200113 is ok :).<br>
<a href="https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20200113&groupid=32" class="external">https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20200113&groupid=32</a><br>
Looks like really wrong setup of HDD_1 was causing this problem.<br>
Let's wait few more builds to be sure.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2722072020-01-17T10:18:03Zpvorelpetr.vorel@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>Builds 20200114 and 20200115 are also ok. + Problem was really just on intel, which was the only one affected by wrong setup => fixed.</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2722102020-01-17T10:19:17Zjlausuchjalausuch@suse.com
<ul></ul><p>Good to hear! Thanks a lot</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2732572020-01-21T19:53:09Zpvorelpetr.vorel@suse.com
<ul></ul><p>Tests fails again: <a href="https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20200119&groupid=32" class="external">https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20200119&groupid=32</a></p>
<p>But this time it's something else <a href="https://openqa.opensuse.org/tests/1150521#" class="external">https://openqa.opensuse.org/tests/1150521#</a></p>
<p>[2020-01-21T16:12:06.0882 CET] [info] +++ setup notes +++<br>
[2020-01-21T16:12:06.0882 CET] [info] Start time: 2020-01-21 15:12:06<br>
[2020-01-21T16:12:06.0882 CET] [info] Running on openqaworker1:7 (Linux 4.12.14-lp151.28.36-default #1 SMP Fri Dec 6 13:50:27 UTC 2019 (8f4a495) x86_64)<br>
[2020-01-21T16:12:06.0890 CET] [info] Downloading <a href="mailto:opensuse-Tumbleweed-x86_64-20200119-DVD@64bit-with-ltp.qcow2">opensuse-Tumbleweed-x86_64-20200119-DVD@64bit-with-ltp.qcow2</a>, request #203793 sent to Cache Service<br>
[2020-01-21T16:12:11.0950 CET] [info] Download of <a href="mailto:opensuse-Tumbleweed-x86_64-20200119-DVD@64bit-with-ltp.qcow2">opensuse-Tumbleweed-x86_64-20200119-DVD@64bit-with-ltp.qcow2</a> processed<br>
[2020-01-21T16:12:11.0985 CET] [info] +++ worker notes +++<br>
[2020-01-21T16:12:11.0985 CET] [info] End time: 2020-01-21 15:12:11<br>
[2020-01-21T16:12:11.0985 CET] [info] Result: setup failure<br>
[2020-01-21T16:12:11.0998 CET] [info] Uploading autoinst-log.txt</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2739472020-01-23T14:48:52Zjlausuchjalausuch@suse.com
<ul></ul><p>Looks fine in latest build...</p>
openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2778672020-02-11T07:40:03Zpvorelpetr.vorel@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed" href="/issues/63373">action #63373</a>: [o3][kernel][scheduler][x86_64] Dependent (child) jobs should start after uploading all of parent assets</i> added</li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=2812932020-02-27T09:55:16Zmetanchrubis@suse.com
<ul><li><strong>Target version</strong> changed from <i>445</i> to <i>457</i></li></ul> openQA Tests - action #51743: [openqa] All LTP tests are failing on boot_ltp for openSUSE (o3) on [x86_64]https://progress.opensuse.org/issues/51743?journal_id=3417972020-10-23T13:39:04Zpcervinkapcervinka@suse.com
<ul><li><strong>Target version</strong> changed from <i>457</i> to <i>QE Kernel Done</i></li></ul>