Project

General

Profile

action #105690

Updated by okurz over 2 years ago

## Observation 

 jobs incomplete with `qemu-img: Could not open '/var/lib/libvirt/images/a.img': Could not open '/var/lib/libvirt/images/a.img': No such file or directory`, for instance https://openqa.suse.de/tests/8047803 : https://openqa.suse.de/tests/8047803: 

 ``` 
 [2022-01-28T09:34:25.924322+01:00] [debug] running `nice ionice qemu-img convert -p -O qcow2 /var/lib/libvirt/images/a.img assets_public/sle-12-SP4-s390x-4.12.14-41.1.g82b276a-Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12-with-ltp.qcow2 -c` 
 [2022-01-28T09:34:25.930323+01:00] [info] ::: backend::driver::__ANON__: Driver backend collected unknown process with pid 398978 and exit status: 0 
 [2022-01-28T09:34:25.930519+01:00] [info] ::: backend::driver::__ANON__: Driver backend collected unknown process with pid 399590 and exit status: 0 
 [2022-01-28T09:34:25.934036+01:00] [debug]       (0.00/100%) 
 qemu-img: Could not open '/var/lib/libvirt/images/a.img': Could not open '/var/lib/libvirt/images/a.img': No such file or directory 
  
 [2022-01-28T09:34:25.934973+01:00] [warn] !!! bmwqemu::serialize_state: unable to extract assets: runcmd 'nice ionice qemu-img convert -p -O qcow2 /var/lib/libvirt/images/a.img assets_public/sle-12-SP4-s390x-4.12.14-41.1.g82b276a-Server-DVD-Incidents-Kernel-KOTD@s390x-kvm-sle12-with-ltp.qcow2 -c' failed with exit code 1: '      (0.00/100%) 
 qemu-img: Could not open '/var/lib/libvirt/images/a.img': Could not open '/var/lib/libvirt/images/a.img': No such file or directory 
   ' at /usr/lib/os-autoinst/osutils.pm line 109. 
  	 osutils::runcmd("nice", "ionice", "qemu-img", "convert", "-p", "-O", "qcow2", "/var/lib/libvirt/images/a.img", ...) called at /usr/lib/os-autoinst/backend/svirt.pm line 189 
  	 backend::svirt::do_extract_assets(backend::svirt=HASH(0x100359af2d8), HASH(0x100349611d8)) called at /usr/lib/os-autoinst/backend/driver.pm line 80 
  	 backend::driver::extract_assets(backend::driver=HASH(0x1003621a9f8), HASH(0x100349611d8)) called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Utils.pm line 163 
  	 eval {...} called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Utils.pm line 163 
  	 OpenQA::Isotovideo::Utils::handle_generated_assets(OpenQA::Isotovideo::CommandHandler=HASH(0x100359c7600), 1) called at /usr/bin/isotovideo line 409 
 ``` 

 seems like https://github.com/os-autoinst/os-autoinst/pull/1936 is the culprit. Complete changelog since last good: 

 > * Thu Jan 27 2022 okurz@suse.com 
 > - Update to version 4.6.1643299616.01abba344: 
 > * Add more perl signatures 
 > * Add more perl signatures 
 > * Simplify code for assigning job settings in `create_from_settings` 
 > * Use constant for referring to a job's main settings 
 > * Allow changing job settings via restart API 
 > * Check whether clones created by the restart API take over the group 
 > * Prevent error when restarting jobs with `skip_parents=1` 
 > * Use tidyall instead of custom implementation 
 >  
 > * Tue Jan 25 2022 okurz@suse.com 
 > - Update to version 4.6.1643089984.09669586b: 
 > * Dependency cron 2022-01-22 
 > * Add signatures to OpenQA::App and OpenQA::BuildResults 
 > * templates: Fix wording for the "VNC display number" 
 > * Show number of restarts of a job within info box on details page 
 >  
 > os-autoinst changes: 
 > * Thu Jan 27 2022 okurz@suse.com 
 > - Update to version 4.6.1643273407.65ca16b7: 
 > * svirt: Store vmname early for use after test run 
 >  
 > * Tue Jan 25 2022 okurz@suse.com 
 > - Update to version 4.6.1643061641.d319802b: 
 > * Continue further checks in fullstack test after one fails 
 > * Add OBS workflow 
 > * Simplify base inheritance statement with Mojo::Base everywhere 
 > * Exclude 29-backend-driver.t from OBS checks 
 > * Simplify string concatenation in log.pm 
 > * Extract all log functions into new module "log" 
 > * Add test for defining/starting VM via VMware in svirt backend 
 > * Add test for generating XML file with UEFI loader in svirt backend 
 > * Use tidyall for faster tidying 
 > * Fix single, unnecessary UTF8 character in consoles::VNC 
 > * git subrepo pull (merge) external/os-autoinst-common 


 ## Reproducible 

 Always reproducible on s390x svirt jobs since we have that change in. 

 ## Expected result 

 Last good https://openqa.suse.de/tests/8019621/logfile?filename=autoinst-log.txt 

 showing: 

 ``` 
 [2022-01-23T01:41:54.907320+01:00] [debug] <<< testapi::type_string(string="nice ionice qemu-img convert -p -O qcow2 /var/lib/libvirt/images/openQA-SUT-3a.img /var/lib/libvirt/images/sle-12-SP4-s390x-4.12.14-40.1.g1475601-Server-DVD-Incidents-Kernel-KOTD\@s390x-kvm-sle12-with-ltp.qcow2 && echo OK", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47) 
 [2022-01-23T01:42:02.749003+01:00] [debug] tests/kernel/../shutdown/svirt_upload_assets.pm:60 called svirt_upload_assets::extract_assets -> tests/kernel/../shutdown/svirt_upload_assets.pm:32 called testapi::assert_screen 
 ``` 


 ## Workaround 

 Revert https://github.com/os-autoinst/os-autoinst/pull/1936 and retrigger tests. OSD deployment was rolled back with https://gitlab.suse.de/openqa/osd-deployment/-/jobs/813165

Back