action #40871

Asset removed during job runtime

Added by mlin7442 over 1 year ago. Updated about 1 year ago.

Status:ResolvedStart date:11/09/2018
Priority:NormalDue date:
Assignee:EDiGiacinto% Done:

0%

Category:Feature requests
Target version:Done
Difficulty:
Duration:

Description

Seen this issue in scenario opensuse-Tumbleweed-DVD-x86_64-update_Leap_42.3_kde@64bit https://openqa.opensuse.org/tests/752322 but
it can probably happens in any test.

2018-09-11T12:38:32.0001 CEST] [debug] QEMU: qemu-system-x86_64: terminating on signal 15 from pid 3883 (/usr/bin/isotovideo: backen)
[2018-09-11T12:38:32.0756 CEST] [debug] running /usr/bin/qemu-img create -f qcow2 -b /var/lib/openqa/pool/3/raid/hd0-overlay8 /var/lib/openqa/pool/3/raid/hd0-overlay9 42949672960
[2018-09-11T12:38:32.0779 CEST] [debug] Formatting '/var/lib/openqa/pool/3/raid/hd0-overlay9', fmt=qcow2 size=42949672960 backing_file=/var/lib/openqa/pool/3/raid/hd0-overlay8 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

[2018-09-11T12:38:32.0780 CEST] [debug] running /usr/bin/qemu-img create -f qcow2 -b /var/lib/openqa/pool/3/raid/cd0-overlay8 /var/lib/openqa/pool/3/raid/cd0-overlay9 4341104640
[2018-09-11T12:38:32.0810 CEST] [debug] Formatting '/var/lib/openqa/pool/3/raid/cd0-overlay9', fmt=qcow2 size=4341104640 backing_file=/var/lib/openqa/pool/3/raid/cd0-overlay8 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

[2018-09-11T12:38:32.0811 CEST] [debug] starting: /usr/bin/qemu-system-x86_64 -vga cirrus -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -soundhw ac97 -global isa-fdc.driveA= -m 1536 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d,menu=on,splash-time=5000 -device usb-ehci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :93,share=force-shared -device virtio-serial -chardev socket,path=virtio_console,server,nowait,id=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-overlay9-file,filename=/var/lib/openqa/pool/3/raid/hd0-overlay9,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0-overlay9,file=hd0-overlay9-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0-overlay9,serial=hd0 -blockdev driver=file,node-name=cd0-overlay9-file,filename=/var/lib/openqa/pool/3/raid/cd0-overlay9,cache.no-flush=on -blockdev driver=qcow2,node-name=cd0-overlay9,file=cd0-overlay9-file,cache.no-flush=on -device scsi-cd,id=cd0-device,drive=cd0-overlay9,serial=cd0 -incoming defer
Attempt 0 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 1 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 2 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 3 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 4 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 5 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 6 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 7 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 8 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 9 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 10 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 11 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 12 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 13 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 14 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 15 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 16 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 17 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 18 at /usr/lib/os-autoinst/osutils.pm line 131.
Attempt 19 at /usr/lib/os-autoinst/osutils.pm line 131.
DIE can't open qmp at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 392.

 at /usr/lib/os-autoinst/backend/baseclass.pm line 80.
    backend::baseclass::die_handler('can\'t open qmp at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm l...') called at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 392
    OpenQA::Qemu::Proc::__ANON__() called at /usr/lib/os-autoinst/osutils.pm line 136
    osutils::attempt('HASH(0x8328258)') called at /usr/lib/os-autoinst/OpenQA/Qemu/Proc.pm line 394
    OpenQA::Qemu::Proc::connect_qmp('OpenQA::Qemu::Proc=HASH(0x7ee3980)') called at /usr/lib/os-autoinst/backend/qemu.pm line 433
    backend::qemu::load_snapshot('backend::qemu=HASH(0x7d8a8a0)', 'HASH(0x115e7910)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 75
    backend::baseclass::handle_command('backend::qemu=HASH(0x7d8a8a0)', 'HASH(0x9515e70)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 487
    backend::baseclass::check_socket('backend::qemu=HASH(0x7d8a8a0)', 'IO::Handle=GLOB(0x7e6b1b0)') called at /usr/lib/os-autoinst/backend/qemu.pm line 1004
    backend::qemu::check_socket('backend::qemu=HASH(0x7d8a8a0)', 'IO::Handle=GLOB(0x7e6b1b0)', 0) called at /usr/lib/os-autoinst/backend/baseclass.pm line 246
    eval {...} called at /usr/lib/os-autoinst/backend/baseclass.pm line 156
    backend::baseclass::run_capture_loop('backend::qemu=HASH(0x7d8a8a0)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 129
    backend::baseclass::run('backend::qemu=HASH(0x7d8a8a0)', 13, 16) called at /usr/lib/os-autoinst/backend/driver.pm line 77
    backend::driver::__ANON__('Mojo::IOLoop::ReadWriteProcess=HASH(0x7f81b88)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
    eval {...} called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
    Mojo::IOLoop::ReadWriteProcess::_fork('Mojo::IOLoop::ReadWriteProcess=HASH(0x7f81b88)', 'CODE(0x27b8f28)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 476
    Mojo::IOLoop::ReadWriteProcess::start('Mojo::IOLoop::ReadWriteProcess=HASH(0x7f81b88)') called at /usr/lib/os-autoinst/backend/driver.pm line 79
    backend::driver::start('backend::driver=HASH(0x63176d0)') called at /usr/lib/os-autoinst/backend/driver.pm line 50
    backend::driver::new('backend::driver', 'qemu') called at /usr/bin/isotovideo line 188
    main::init_backend() called at /usr/bin/isotovideo line 244
[2018-09-11T12:38:51.0992 CEST] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
last frame
[2018-09-11T12:38:52.0043 CEST] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2018-09-11T12:38:52.0048 CEST] [debug] QEMU: QEMU emulator version 2.9.1(openSUSE Leap 42.3)
[2018-09-11T12:38:52.0048 CEST] [debug] QEMU: Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
[2018-09-11T12:38:52.0048 CEST] [debug] QEMU: qemu-system-x86_64: -blockdev driver=qcow2,node-name=hd0-overlay9,file=hd0-overlay9-file,cache.no-flush=on: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open backing file: Could not open '/var/lib/openqa/pool/3/opensuse-42.3-x86_64-GM-kde@64bit.qcow2': No such file or directory
[2018-09-11T12:38:52.0048 CEST] [debug] sending magic and exit
[2018-09-11T12:38:52.0049 CEST] [debug] received magic close
[2018-09-11T12:38:52.0049 CEST] [debug] THERE IS NOTHING TO READ 15 4 3
[2018-09-11T12:38:52.0060 CEST] [debug] backend process exited: 0
[2018-09-11T12:38:52.0065 CEST] [debug] commands process exited: 0
[2018-09-11T12:38:53.0065 CEST] [debug] sysread failed: 
syswrite failed Broken pipe at /usr/lib/os-autoinst/myjsonrpc.pm line 38.
    myjsonrpc::send_json('GLOB(0x630d0e0)', 'HASH(0x360e4f0)') called at /usr/lib/os-autoinst/autotest.pm line 282
    autotest::query_isotovideo('backend_reset_console', 'HASH(0x6c73df8)') called at /usr/lib/os-autoinst/basetest.pm line 633
    basetest::rollback_activated_consoles('consoletest_finish=HASH(0x5ef51d0)') called at /usr/lib/os-autoinst/autotest.pm line 341
    autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 183
    eval {...} called at /usr/lib/os-autoinst/autotest.pm line 183
    autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 236
    autotest::__ANON__('Mojo::IOLoop::ReadWriteProcess=HASH(0x62f7590)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
    eval {...} called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
    Mojo::IOLoop::ReadWriteProcess::_fork('Mojo::IOLoop::ReadWriteProcess=HASH(0x62f7590)', 'CODE(0x63167d0)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/IOLoop/ReadWriteProcess.pm line 476
    Mojo::IOLoop::ReadWriteProcess::start('Mojo::IOLoop::ReadWriteProcess=HASH(0x62f7590)') called at /usr/lib/os-autoinst/autotest.pm line 237
    autotest::start_process() called at /usr/bin/isotovideo line 242
[2018-09-11T12:38:53.0078 CEST] [debug] [autotest] process exited
: 0


Related issues

Related to openQA Project - action #39980: Cache locks assets when worker dies in critical section Resolved 02/10/2018
Related to openQA Project - action #55514: job failed to lastgood snapshot "Could not open backing f... Rejected 14/08/2019

History

#1 Updated by mkittler over 1 year ago

  • Description updated (diff)

#2 Updated by coolo over 1 year ago

  • Target version set to Current Sprint

https://openqa.suse.de/tests/2057213/ shows the same thing :(

#3 Updated by coolo over 1 year ago

The job is missing an asset - and while it finished 4 minutes ago, it's gone now. So I suspect the worker cache is not only problematic during download :(

#4 Updated by coolo over 1 year ago

  • Subject changed from isotovideo broken after loads lastgood to Asset removed during job runtime

#5 Updated by coolo over 1 year ago

  • Status changed from New to In Progress
  • Assignee set to EDiGiacinto

this is part of the worker story - and already worked on

#6 Updated by EDiGiacinto over 1 year ago

  • Related to action #39980: Cache locks assets when worker dies in critical section added

#7 Updated by EDiGiacinto over 1 year ago

  • Status changed from In Progress to Feedback

https://github.com/os-autoinst/openQA/pull/1783 has been merged, setting to Feedback

#8 Updated by coolo about 1 year ago

  • Status changed from Feedback to Resolved
  • Target version changed from Current Sprint to Done

#9 Updated by okurz 6 months ago

  • Related to action #55514: job failed to lastgood snapshot "Could not open backing file: Could not open backing file: Could not open …" added

Also available in: Atom PDF