Project

General

Profile

action #75259

Updated by okurz over 3 years ago

## Observation 
 Currently, all powerpc tests end with result incomplete 

 E.g https://openqa.opensuse.org/tests/1448500 

 From the log: 

 [2020-10-25T08:55:26.0807 UTC] [info] +++ setup notes +++ 
 [2020-10-25T08:55:26.0807 UTC] [info] Running on power8:6 (Linux 5.3.18-lp152.47-default #1 SMP Thu Oct 15 16:05:25 UTC 2020 (41f7396) ppc64le) 
 [2020-10-25T08:55:26.0810 UTC] [debug] Found ASSET_256, caching openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256 
 [2020-10-25T08:55:26.0812 UTC] [info] Downloading openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256, request #37952 sent to Cache Service 
 [2020-10-25T08:55:31.0853 UTC] [info] Download of openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256 processed: 
 [info] [#37952] Cache size of "/var/lib/openqa/cache" is 600GiB, with limit 600GiB 
 [info] [#37952] Downloading "openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256" from "http://openqa1-opensuse/tests/1448500/asset/other/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256" 
 [info] [#37952] Content of "/var/lib/openqa/cache/openqa1-opensuse/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso.sha256" has not changed, updating last use 

 [2020-10-25T08:55:31.0856 UTC] [debug] Found ISO, caching openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso 
 [2020-10-25T08:55:31.0858 UTC] [info] Downloading openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso, request #37957 sent to Cache Service 
 [2020-10-25T08:55:36.0899 UTC] [info] Download of openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso processed: 
 [info] [#37957] Cache size of "/var/lib/openqa/cache" is 600GiB, with limit 600GiB 
 [info] [#37957] Downloading "openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso" from "http://openqa1-opensuse/tests/1448500/asset/iso/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso" 
 [info] [#37957] Content of "/var/lib/openqa/cache/openqa1-opensuse/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso" has not changed, updating last use 

 [2020-10-25T08:55:36.0902 UTC] [info] Rsync from 'rsync://openqa1-opensuse/tests' to '/var/lib/openqa/cache/openqa1-opensuse', request #37961 sent to Cache Service 
 [2020-10-25T08:55:41.0954 UTC] [info] Output of rsync: 
 [info] [#37961] Calling: rsync -avHP rsync://openqa1-opensuse/tests/ --delete /var/lib/openqa/cache/openqa1-opensuse/tests/ 
 receiving incremental file list 

 sent 1,713 bytes    received 1,099,636 bytes    734,232.67 bytes/sec 
 total size is 4,532,146,856    speedup is 4,115.09 

 [2020-10-25T08:55:41.0954 UTC] [info] Finished to rsync tests 
 [2020-10-25T08:55:41.0958 UTC] [debug] +++ worker notes +++ 
 [2020-10-25T08:55:42.377 UTC] [debug] Current version is 4.6.1603469837.f54bdeab [interface v20] 
 [2020-10-25T08:55:42.384 UTC] [debug] git hash in /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse: c12d1da2e511d1cb87aeb302b752bc035a2c8c60 
 [2020-10-25T08:55:42.560 UTC] [debug] usingenv DESKTOP=kde 
 [2020-10-25T08:55:42.560 UTC] [debug] usingenv DISTRI=opensuse 
 [2020-10-25T08:55:42.560 UTC] [debug] usingenv QEMUCPU=host 
 [2020-10-25T08:55:42.560 UTC] [debug] usingenv QEMUCPUS=4 
 [2020-10-25T08:55:42.560 UTC] [debug] usingenv INSTLANG=en_US 
 [2020-10-25T08:55:42.560 UTC] [debug] usingenv DVD=1 
 [2020-10-25T08:55:42.560 UTC] [debug] usingenv KDE=1 
 [2020-10-25T08:55:42.560 UTC] [debug] usingenv ISO=/var/lib/openqa/pool/6/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso 
 [2020-10-25T08:55:42.561 UTC] [debug] usingenv ISO_MAXSIZE=4700372992 
 [2020-10-25T08:55:42.561 UTC] [debug] usingenv NOIMAGES=1 
 [2020-10-25T08:55:42.561 UTC] [debug] scheduling mediacheck tests/installation/mediacheck.pm 
 [2020-10-25T08:55:42.567 UTC] [info] cmdsrv: daemon reachable under http://*:20063/qL0cvoaoKR5zOTFr/ 
 [2020-10-25T08:55:42.568 UTC] [debug] git hash in /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles: 5482b32dfe22ba8e0376db50c26e07cc01ee56fb 
 [2020-10-25T08:55:42.568 UTC] [debug] init needles from /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles 
 [2020-10-25T08:55:42.569 UTC] [info] Listening at "http://[::]:20063" 
 Web application available at http://[::]:20063 
 [2020-10-25T08:55:42.884 UTC] [debug] inst-console-20200224 contains inst-console twice 
 [2020-10-25T08:55:43.715 UTC] [debug] loaded 8201 needles 
 [2020-10-25T08:55:43.862 UTC] [debug] Blocking SIGTERM 
 [2020-10-25T08:55:43.884 UTC] [debug] Unblocking SIGTERM 
 [2020-10-25T08:55:43.981 UTC] [debug] Blocking SIGTERM 
 19934: channel_out 15, channel_in 14 
 [2020-10-25T08:55:44.000 UTC] [debug] Unblocking SIGTERM 
 19982: cmdpipe 13, rsppipe 16 
 [2020-10-25T08:55:44.001 UTC] [debug] started mgmt loop with pid 19982 
 remove_tree qemuscreenshot 
 [2020-10-25T08:55:44.025 UTC] [debug] qemu version detected: 4.2.1 
 [2020-10-25T08:55:44.026 UTC] [debug] running /usr/bin/chattr -f +C /var/lib/openqa/pool/6/raid 
 [2020-10-25T08:55:44.032 UTC] [debug] running /usr/bin/qemu-img info --output=json /var/lib/openqa/pool/6/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso 
 [2020-10-25T08:55:44.039 UTC] [debug] running /usr/bin/qemu-img create -f qcow2 /var/lib/openqa/pool/6/raid/hd0 20G 
 [2020-10-25T08:55:44.076 UTC] [debug] Formatting '/var/lib/openqa/pool/6/raid/hd0', fmt=qcow2 size=21474836480 cluster_size=65536 lazy_refcounts=off refcount_bits=16 
 [2020-10-25T08:55:44.076 UTC] [debug] running /usr/bin/qemu-img create -f qcow2 -b /var/lib/openqa/pool/6/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso /var/lib/openqa/pool/6/raid/cd0-overlay0 3745804288 
 [2020-10-25T08:55:44.119 UTC] [debug] Formatting '/var/lib/openqa/pool/6/raid/cd0-overlay0', fmt=qcow2 size=3745804288 backing_file=/var/lib/openqa/pool/6/openSUSE-Tumbleweed-DVD-ppc64le-Snapshot20201024-Media.iso cluster_size=65536 lazy_refcounts=off refcount_bits=16 
 [2020-10-25T08:55:44.120 UTC] [debug] starting: /usr/bin/qemu-system-ppc64 -g 1024x768 -vga std -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.driveA= -m 4096 -machine usb=off -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d -device nec-usb-xhci -device usb-tablet -device usb-kbd -smp 4 -enable-kvm -no-shutdown -vnc :96,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-file,filename=/var/lib/openqa/pool/6/raid/hd0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0,file=hd0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0,serial=hd0 -blockdev driver=file,node-name=cd0-overlay0-file,filename=/var/lib/openqa/pool/6/raid/cd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=cd0-overlay0,file=cd0-overlay0-file,cache.no-flush=on -device scsi-cd,id=cd0-device,drive=cd0-overlay0,serial=cd0 
 [2020-10-25T08:55:44.123 UTC] [debug] Waiting for 0 attempts 
 [2020-10-25T08:55:44.185 UTC] [debug] Waiting for 1 attempts 
 [2020-10-25T08:55:44.185 UTC] [debug] Backend process died, backend errors are reported below in the following lines: 
 QEMU terminated before QMP connection could be established at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 444. 

 [2020-10-25T08:55:44.185 UTC] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json 
 [2020-10-25T08:55:44.186 UTC] [debug] Passing remaining frames to the video encoder 
 [2020-10-25T08:55:44.187 UTC] [debug] Waiting for video encoder to finalize the video 
 [2020-10-25T08:55:44.187 UTC] [debug] The built-in video encoder (pid 20002) terminated 
 [2020-10-25T08:55:44.188 UTC] [debug] QEMU: QEMU emulator version 4.2.1 (openSUSE Leap 15.2) 
 [2020-10-25T08:55:44.188 UTC] [debug] QEMU: Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers 
 [2020-10-25T08:55:44.188 UTC] [debug] QEMU: Unknown host! 
 [2020-10-25T08:55:44.188 UTC] [debug] QEMU: Unknown host! 
 [2020-10-25T08:55:44.188 UTC] [debug] QEMU: Unknown host! 
 [2020-10-25T08:55:44.188 UTC] [debug] QEMU: qemu-system-ppc64: Requested safe cache capability level not supported by kvm, try appending -machine cap-cfpc=broken 

 ## Problem 

 This very likely happened due to the upgrade of the worker host to openSUSE Leap 15.2 in #73189 

 ## Suggestions 
 * First consider rolling back the installation or maybe just the qemu version 
 * Try the suggestion in the log file: "try appending -machine cap-cfpc=broken", to the qemu command line in jobs, then add that maybe in the openQA "machine" configuration? 
 * Consider including the necessary setting in os-autoinst itself

Back