Need to provide workaround for getting logs from system without network
CUrrently we face from time to time with scenario where network is broken and we can do much for analyze issues which happening on broken system.
Mainly it is about upload_logs function. Idea is to make upload_logs network agnostic.
#2 Updated by asmorodskyi over 4 years ago
I know that it limits us to qemu , but still what you think about :
- Test fails
- In post_fail_hook there is attempt to upload_logs which fails because there is no network
- This trigger last standing post fail hook which running not in context of the test but in context of worker itself
- Worker shutdown qemu and then trying to do something like this https://www.jamescoyle.net/how-to/1818-access-a-qcow2-virtual-disk-image-from-the-host.
What do you think ?
#4 Updated by coolo over 4 years ago
If you have test suites that need to work without network, provide a shared directory through vfat:
http://qemu-buch.de/de/index.php?title=QEMU-KVM-Buch/_Speichermedien/_Virtuelle_FAT-Festplatten and save the logs there. and they will be emulated as floppies in qemu.
#8 Updated by okurz over 3 years ago
Comment from AdamWill in #45086:
Sometimes, a test failure occurs because there's no network connection in the SUT. Then we hit our post-fail hook, which probably tries to upload logs for diagnosing the problem...over the network...
I'm doing a sorta hacky workaround for this in the Fedora tests: have the post-fail hook check if it can ping the host IP, and if it can't, try and dump the contents of text logs via the serial console. That seems at least better than nothing.
It occurred to me that it may be interesting to do a more generalized implementation of this in os-autoinst - perhaps as a new command similar to upload_logs, or implement it within upload_logs based on whether it can ping the host IP, something like that? Does anyone think this sounds like a useful idea?
#9 Updated by AdamWill over 3 years ago
FWIW, this is my dirty workaround:
it works. Of course, it's no use for non-text files, and it's a bit 'messy' (you get the output all mixed in with the script_run exit code lines and stuff). And I dunno what'd happen if your post_fail_hook was actually using the serial console (not a VT)...
Using floppy / USB storage emulation somehow seems like a good idea, not sure if we could actually 'help' with that in os-autoinst or openQA, or if it'd just have to be something implemented in the distro test suites...