action #49247
closed[y] Fix MAKETESTSNAPSHOTS feature of openQA
0%
Description
Motivation¶
Current version of openQA has following behavior:
After run to create snapshots is done, one run with SKIPTO will work flawlessly.
Second run with SKIPTO will fail.
Here is the documentation of the feature: https://github.com/os-autoinst/openQA/blob/master/docs/WritingTests.asciidoc#using-snapshots-to-speed-up-development-of-tests
Acceptance criteria¶
- MAKETESTSNAPSHOTS feature works as before
[2019-03-04T17:41:21.327 CET] [debug] Snapshots are supported
[2019-03-04T17:41:21.329 CET] [debug] skipping installation-isosize
[2019-03-04T17:41:21.331 CET] [debug] skipping installation-bootloader
[2019-03-04T17:41:21.332 CET] [debug] skipping installation-welcome
[2019-03-04T17:41:21.334 CET] [debug] skipping installation-accept_license
[2019-03-04T17:41:21.335 CET] [debug] skipping installation-scc_registration
[2019-03-04T17:41:21.336 CET] [debug] skipping installation-addon_products_sle
[2019-03-04T17:41:21.338 CET] [debug] skipping installation-system_role
[2019-03-04T17:41:21.339 CET] [debug] skipping installation-partitioning
[2019-03-04T17:41:21.340 CET] [debug] skipping installation-partitioning_raid
[2019-03-04T17:41:21.341 CET] [debug] skipping installation-partitioning_finish
[2019-03-04T17:41:21.342 CET] [debug] skipping installation-installer_timezone
[2019-03-04T17:41:21.342 CET] [debug] skipping installation-hostname_inst
[2019-03-04T17:41:21.343 CET] [debug] skipping installation-user_settings
[2019-03-04T17:41:21.344 CET] [debug] skipping installation-user_settings_root
[2019-03-04T17:41:21.344 CET] [debug] skipping installation-installation_overview
[2019-03-04T17:41:21.345 CET] [debug] skipping installation-disable_grub_timeout
[2019-03-04T17:41:21.346 CET] [debug] skipping installation-start_install
[2019-03-04T17:41:21.347 CET] [debug] skipping installation-await_install
[2019-03-04T17:41:21.348 CET] [debug] Loading a VM snapshot installation-logs_from_installation_system
[2019-03-04T17:41:21.349 CET] [debug] Loading snapshot (Current VM state is inmigrate).
[2019-03-04T17:41:21.349 CET] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2019-03-04T17:41:23.380 CET] [debug] QEMU: qemu-system-x86_64: terminating on signal 15 from pid 27457 (/usr/bin/isotovideo: backen)
[2019-03-04T17:41:23.383 CET] [debug] starting: /usr/bin/qemu-system-x86_64 -vga cirrus -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -soundhw ac97 -global isa-fdc.driveA= -m 1024 -cpu qemu64 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d,menu=on,splash-time=5000 -device usb-ehci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :91,share=force-shared -device virtio-serial -chardev socket,path=virtio_console,server,nowait,id=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -incoming defer
Attempt 0 at /usr/lib/os-autoinst/osutils.pm line 130.
Attempt 1 at /usr/lib/os-autoinst/osutils.pm line 130.
Attempts terminated! at /usr/lib/os-autoinst/osutils.pm line 136.
[2019-03-04T17:41:25.396 CET] [debug] qmpsocket 14
[2019-03-04T17:41:25.404 CET] [debug] EVENT {"data":{"status":"setup"},"event":"MIGRATION","timestamp":{"microseconds":404349,"seconds":1551717685}}
[2019-03-04T17:41:30.322 CET] [debug] Backend process died, backend errors are reported below in the following lines:
Can't syswrite(IO::Socket::UNIX=GLOB(0x55ca6eec1318), <BUFFER>): Broken pipe at /usr/lib/os-autoinst/backend/qemu.pm line 956
[2019-03-04T17:41:30.323 CET] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
last frame
[2019-03-04T17:41:30.464 CET] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2019-03-04T17:41:30.464 CET] [debug] QEMU: QEMU emulator version 3.1.0 (openSUSE Tumbleweed)
[2019-03-04T17:41:30.464 CET] [debug] QEMU: Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers
[2019-03-04T17:41:30.465 CET] [debug] QEMU: qemu-system-x86_64: Unknown savevm section or instance '0000:00:07.0/virtio-scsi' 0. Make sure that your current VM setup matches your saved VM setup, including any hotplugged devices
[2019-03-04T17:41:30.465 CET] [debug] QEMU: qemu-system-x86_64: load of migration failed: Invalid argument
[2019-03-04T17:41:30.466 CET] [debug] sending magic and exit
[2019-03-04T17:41:30.466 CET] [debug] received magic close
[2019-03-04T17:41:30.466 CET] [debug] THERE IS NOTHING TO READ 15 4 3
[2019-03-04T17:41:30.471 CET] [debug] backend process exited: 0
[2019-03-04T17:41:30.474 CET] [debug] commands process exited: 0
[2019-03-04T17:41:31.475 CET] [debug] sysread failed:
[2019-03-04T17:41:31.475 CET] [debug] ||| starting logs_from_installation_system tests/installation/logs_from_installation_system.pm
syswrite failed Broken pipe at /usr/lib/os-autoinst/myjsonrpc.pm line 40.
myjsonrpc::send_json(GLOB(0x55ca6b057208), HASH(0x55ca6b0979a0)) called at /usr/lib/os-autoinst/autotest.pm line 313
autotest::query_isotovideo("set_current_test", HASH(0x55ca6ca9f360)) called at /usr/lib/os-autoinst/autotest.pm line 181
autotest::set_current_test(logs_from_installation_system=HASH(0x55ca6ca05ce0)) called at /usr/lib/os-autoinst/basetest.pm line 272
basetest::start(logs_from_installation_system=HASH(0x55ca6ca05ce0)) called at /usr/lib/os-autoinst/autotest.pm line 350
autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 214
eval {...} called at /usr/lib/os-autoinst/autotest.pm line 214
autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 267
autotest::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x55ca6c93f340)) called at /usr/lib/perl5/vendor_perl/5.28.1/Mojo/IOLoop/ReadWriteProcess.pm line 325
eval {...} called at /usr/lib/perl5/vendor_perl/5.28.1/Mojo/IOLoop/ReadWriteProcess.pm line 325
Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x55ca6c93f340), CODE(0x55ca6c85de38)) called at /usr/lib/perl5/vendor_perl/5.28.1/Mojo/IOLoop/ReadWriteProcess.pm line 476
Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x55ca6c93f340)) called at /usr/lib/os-autoinst/autotest.pm line 268
autotest::start_process() called at /usr/bin/isotovideo line 251
[2019-03-04T17:41:31.489 CET] [debug] [autotest] process exited: 0
[2019-03-04T17:41:32.490 CET] [debug] isotovideo failed
Updated by riafarov about 5 years ago
- Blocks action #36090: Save snapshots by name added
Updated by okurz about 5 years ago
Have you run the second run with MAKETESTSNAPSHOTS=1 as well? Otherwise the second run will delete all snapshots after use (I think).
Updated by riafarov about 5 years ago
okurz wrote:
Have you run the second run with MAKETESTSNAPSHOTS=1 as well? Otherwise the second run will delete all snapshots after use (I think).
Hmm, no, without it, but it should not remove with --no-clean-up flag set for workers, but yeah, makes sense to check, maybe it works this way.
Updated by riafarov about 5 years ago
- Due date changed from 2019-04-23 to 2019-05-07
Updated by riafarov about 5 years ago
- Description updated (diff)
- Status changed from New to Workable
- Estimated time set to 13.00 h
Updated by riafarov about 5 years ago
- Due date changed from 2019-05-07 to 2019-05-21
Updated by JERiveraMoya almost 5 years ago
- Due date changed from 2019-05-21 to 2019-06-04
Updated by JERiveraMoya almost 5 years ago
- Status changed from Workable to In Progress
- Assignee set to JERiveraMoya
Updated by JERiveraMoya almost 5 years ago
I found following behavior:
http://rivera-workstation.suse.cz/tests/2760 (MAKETESTSNAPSHOTS) OK
http://rivera-workstation.suse.cz/tests/2762 (1st SKIPTO) OK
http://rivera-workstation.suse.cz/tests/2763 (2nd SKIPTO) OK
http://rivera-workstation.suse.cz/tests/2764 (3nd SKIPTO previous test) OK
http://rivera-workstation.suse.cz/tests/2765 (4nd SKIPTO same initial test which is after) Fail loading a VM snapshot:
[2019-05-22T15:41:22.881 CEST] [debug] sysread failed:
Use of uninitialized value $command in string eq at /usr/lib/os-autoinst/autotest.pm line 211.
[2019-05-22T15:41:22.881 CEST] [debug] ||| starting system_role tests/installation/system_role.pm
syswrite failed Broken pipe at /usr/lib/os-autoinst/myjsonrpc.pm line 40.
myjsonrpc::send_json(GLOB(0x55707e9ff578), HASH(0x55707ea4fc78)) called at /usr/lib/os-autoinst/autotest.pm line 320
autotest::query_isotovideo("set_current_test", HASH(0x5570809ae930)) called at /usr/lib/os-autoinst/autotest.pm line 181
autotest::set_current_test(system_role=HASH(0x5570807d30c0)) called at /usr/lib/os-autoinst/basetest.pm line 273
basetest::start(system_role=HASH(0x5570807d30c0)) called at /usr/lib/os-autoinst/autotest.pm line 357
autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 221
eval {...} called at /usr/lib/os-autoinst/autotest.pm line 221
autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 274
autotest::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x5570806e76b8)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 325
eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 325
Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x5570806e76b8), CODE(0x55707f5fd668)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 476
Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x5570806e76b8)) called at /usr/lib/os-autoinst/autotest.pm line 275
autotest::start_process() called at /usr/bin/isotovideo line 304
http://rivera-workstation.suse.cz/tests/2766 (5nd SKIPTO previous test) OK
http://rivera-workstation.suse.cz/tests/2768 (6nd SKIPTO previous test to previous test) OK
Was not this bug/problem/behavior already there when the feature was working? I believe so, that we can only go back to snapshots which are located before the one where is performed first SKIPTO. In my opinion and using my local setup (recently updated Leap 15!) seems to works the same than before without using of MAKETESTSNAPSHOTS twice.
Updated by JERiveraMoya almost 5 years ago
Today it succeeded as well with a complete successful run (instead of one canceled): http://rivera-workstation.suse.cz/tests/2769 (MAKETESTSNAPSHOTS) and http://rivera-workstation.suse.cz/tests/2771 (SKIPTO)
I noticed that trying to go a test after (what yesterday was failing) now works (but today I have previously do one full run with MAKETESTSNAPSHOTS as described in previous sentence with same worker and different test suite). Nevertheless the same problem is reproducible when trying to go to a test scheduled after: http://rivera-workstation.suse.cz/tests/2773, so same behavior, the only difference is to run MAKETESTSNAPSHOT in the middle.
The feature seems to be useful, it would be good if someone else could try out, maybe it is just about to have up-to-date os-autoinst.
Updated by JERiveraMoya almost 5 years ago
- Status changed from In Progress to Feedback
Updated by riafarov almost 5 years ago
so, I've tried first with btrfs test suite and faced the issue, but it worked perfectly with RAID0. From settings the difference is NUMDISKS=4 and HDDSIZE (40 gb vs 20 gb). Taking deeper look.
Updated by riafarov almost 5 years ago
So I got quite inconsistent results, might be due to polluted workers directory. So btrfs scenario also worked 3 times in a row, so not sure about the steps which caused different behavior.
Updated by riafarov almost 5 years ago
- Status changed from Feedback to Resolved
Feature worked for 3 of us.