openQA Project (public)

As we have so much fun with migrate command: https://progress.opensuse.org/issues/19390

Currently we are using it to pipe an entire memory dump into bzip then redirecting that to a file through the shell. Despite the fact this is the documented way of doing it, I don't think it is the most common way. Libvirt is passing a file descriptor using SCM rights (with the option of doing an incremental dump which is useful for snapshots), so it is possible that if we do it the same way then we will avoid some bottleneck in the exec code. Although I don't see what could be wrong with the exec code in QEMU.

Also libvirt calls migrate_set_speed with the largest value possible before taking a snapshot. I think it is possible that QEMU automatically throttles migrations to prevent them from using all the bandwidth so we should probably do the same when migrating to a file.

OK, so migrating to and from a file works, however it requires restarting QEMU. That probably means refactoring the start_qemu os-autoinst code quite a lot. Alternatively it should be possible patch QEMU to allow incoming migrations without a restart.

Before doing that it is probably a good idea to try making the memory dump reliable. If it suffers from the same problems as savevm (after applying all the fixes I can think of) then there are probably better things to be doing.

It should also be possible to enable performance monitoring during savevm and memory dump which might help reveal where the problem is.

On a semi-related note it looks like snapshots are adding 100GB to the image size for the image published by install_ltp

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4283

From ongoing discussions on the QEMU mailing list it appears various people are working on related problems and solutions for snapshots. I'm not sure I understand all of the problems and solutions, but I think that we should be able to patch QEMU so that it allows an incoming migration without being restarted. This might turn up some bugs or problems which were previously hidden in QEMU (maybe not a bad thing), but it is probably the least intrusive change. It seems to be worth further investigation. Some people appear to be working on solutions which look better than this, but they may take a long time.

It appears that incoming migrations do work in a non-pristine QEMU with a few minor changes, the patch is fairly simple so far: https://github.com/richiejp/qemu/commit/788156d104079ef0deb9e48048a7995f1995dc22. However I wonder what kind of edge cases there are and what run states should be considered a valid starting point.

Well, find out what upstream tells you about the patch. In the past, this turned into a dead end often enough :(

The mail archive link: http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg06782.html

Loading a VM snapshot/migration into a QEMU instance which has already ran a VM appears to work, but I am concerned about the following:

1) There is nothing stopping devices from reading or writing memory during the migration if they do not follow the memory or bus APIs.
2) Many devices hold state outside of guest RAM and it is down to the device to correctly load new state. It is not clear if all devices overwrite existing state correctly.
3) It is not clear that the vCPUs are all guaranteed to have stopped by the time the migration starts.

I have not found any instances of (1) or (2), but QEMU has a large number of complex devices so auditing it would be difficult. It seems that 'loadvm' usually works, but we can not be certain that all devices have a consistent state. If an error in the guest kernel is caused by a device with inconsistent state it may be almost impossible to determine what caused the bug. So while I am sure QEMU can be patched to allow incoming migrations without '-defer' and it would work most of the time (and be quite useful for some people), we really should restart QEMU before loading a VM.