action #36457
closedImplement serial terminal console for svirt backend
0%
Description
Probably not easy to implement, but kernel & networking (and other console tests) would benefit from it:
- Being much faster
- Having text based console logs (instead of screenshots of text)
Files
Updated by pvorel over 6 years ago
- Related to action #32599: [tools][openqa][ppc64le][s390x_zkvm] Kernel log missing from serial0.txt added
Updated by coolo over 6 years ago
To begin with: there is no zkvm backend - and it's not ppc64le related. And I thought you're heading for hardware tests anyway.
Updated by pvorel over 6 years ago
- Subject changed from [tools][openqa][ppc64le][s390x_zkvm] Implement virtio console for zkvm backend to [tools][openqa][s390x_zkvm] Implement virtio console for svirt backend
Updated by coolo about 6 years ago
- Has duplicate action #41840: [sle][s390x][s390-kvm][tools][ltp] Enable virtio_terminal console (VIRTIO_CONSOLE) on svirt backend added
Updated by coolo about 6 years ago
- Subject changed from [tools][openqa][s390x_zkvm] Implement virtio console for svirt backend to Implement virtio console for svirt backend
- Category set to 132
- Priority changed from Normal to High
- Target version set to Ready
Updated by mkittler about 6 years ago
- Assignee set to mkittler
So far I'm just looking into how I can setup a development environment for the svirt backend. Despite the initial topic of the ticket, the svirt backend looks to me architecture neutral. Maybe I could even just ssh to my own machine where I have virsh and qemu installed. For me it would be easier not having to deal with s390pb (or whatever I could use for the jump-host?).
When I'm really starting to work on this I'm going to set the status to in progress.
Updated by coolo about 6 years ago
Yes, you can just use localhost as svirt host
Updated by coolo about 6 years ago
- Target version changed from Ready to Current Sprint
Updated by mkittler about 6 years ago
- Status changed from New to In Progress
I managed to get jobs running with the svirt backend locally using a worker config like I documented here. If you like I can also polish and upstream this documentation.
I also managed to enable virtio in qemu when started via libvirtd. To test this I adjusted the test distribution.
Right now it can not connect to the virtio console because there are issues getting the path of the socket right:
[2018-10-25T15:47:40.0284 CEST] [debug] <<< consoles::virtio_terminal::open_socket(socket_path='/hdd/openqa-devel/openqa/pool/2/virtio_console')
DIE Can't connect($fh, '�/hdd/openqa-devel/openqa/pool/2/virtio_console��������������������������������������������������������������'): Connection refused at /hdd/openqa-devel/repos/os-autoinst/consoles/virtio_terminal.pm line 127
at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 80.
backend::baseclass::die_handler(autodie::exception=HASH(0x5563a74c7270)) called at (eval 178) line 33
consoles::virtio_terminal::__ANON__(GLOB(0x5563a74d4f70), "\x{1}\x{0}/hdd/openqa-devel/openqa/pool/2/virtio_console\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}"...) called at /hdd/openqa-devel/repos/os-autoinst/consoles/virtio_terminal.pm line 127
consoles::virtio_terminal::open_socket(consoles::virtio_terminal=HASH(0x5563a6fa4e38)) called at /hdd/openqa-devel/repos/os-autoinst/consoles/virtio_terminal.pm line 136
consoles::virtio_terminal::activate(consoles::virtio_terminal=HASH(0x5563a6fa4e38)) called at /hdd/openqa-devel/repos/os-autoinst/consoles/console.pm line 85
consoles::console::select(consoles::virtio_terminal=HASH(0x5563a6fa4e38)) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 529
backend::baseclass::select_console(backend::svirt=HASH(0x5563a7bc1ae8), HASH(0x5563a7508d38)) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 75
backend::baseclass::handle_command(backend::svirt=HASH(0x5563a7bc1ae8), HASH(0x5563a74d1370)) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 487
backend::baseclass::check_socket(backend::svirt=HASH(0x5563a7bc1ae8), IO::Handle=GLOB(0x5563a66a96a8), 0) called at /hdd/openqa-devel/repos/os-autoinst/backend/svirt.pm line 236
backend::svirt::check_socket(backend::svirt=HASH(0x5563a7bc1ae8), IO::Handle=GLOB(0x5563a66a96a8), 0) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 246
eval {...} called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 156
backend::baseclass::run_capture_loop(backend::svirt=HASH(0x5563a7bc1ae8)) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 129
backend::baseclass::run(backend::svirt=HASH(0x5563a7bc1ae8), 13, 16) called at /hdd/openqa-devel/repos/os-autoinst/backend/driver.pm line 92
backend::driver::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x5563a785b458)) called at /usr/lib/perl5/vendor_perl/5.26.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
eval {...} called at /usr/lib/perl5/vendor_perl/5.26.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x5563a785b458), CODE(0x5563a776fc40)) called at /usr/lib/perl5/vendor_perl/5.26.2/Mojo/IOLoop/ReadWriteProcess.pm line 476
Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x5563a785b458)) called at /hdd/openqa-devel/repos/os-autoinst/backend/driver.pm line 94
backend::driver::start(backend::driver=HASH(0x5563a676e320)) called at /hdd/openqa-devel/repos/os-autoinst/backend/driver.pm line 50
backend::driver::new("backend::driver", "svirt") called at /hdd/openqa-devel/repos/os-autoinst/isotovideo line 184
main::init_backend() called at /hdd/openqa-devel/repos/os-autoinst/isotovideo line 249
However, I can manually connect to the socket. To do that I've just paused the test using the developer mode before it would attempt to connect and executed the Perl code myself:
sudo perl -I. -de1
...
DB<1> use consoles::virtio_terminal;
DB<2> $term = consoles::virtio_terminal->new();
DB<3> print($term->socket_path);
/hdd/openqa-devel/repos/os-autoinst/virtio_console
DB<4> $term->open_socket();
[2018-10-25T15:19:03.0360 CEST] [debug] <<< consoles::virtio_terminal::open_socket(socket_path='/hdd/openqa-devel/repos/os-autoinst/virtio_console')
Could not find /hdd/openqa-devel/repos/os-autoinst/virtio_console at (eval 108)[/usr/lib/perl5/5.26.2/perl5db.pl:738] line 2.
DB<5> $term->{socket_path} = '/var/lib/libvirt/qemu/channel/target/domain-4-openQA-SUT-1/org.openqa.console.virtio_console';
DB<6> $term->open_socket();
[2018-10-25T15:19:09.0031 CEST] [debug] <<< consoles::virtio_terminal::open_socket(socket_path='/var/lib/libvirt/qemu/channel/target/domain-4-openQA-SUT-1/org.openqa.console.virtio_console')
<no error>
So there are multiple things TODO:
- Tell libvirt to create the socket at a specific location or try to detect it at its default location.
- Make the virtio socket on the svirt host available on the worker where the test is executed (eg. SSH tunnel?).
Updated by coolo about 6 years ago
- Subject changed from Implement virtio console for svirt backend to Implement serial terminal console for svirt backend
Forget about qemu and it's virtio - I thought I made myself clear there. They want this for all backends
Updated by mkittler about 6 years ago
Ok, so I guess I'll have to do it like described here: https://help.ubuntu.com/community/KVM/Access
So the modification to the XML will become even easier but I will need to configure the guest. And I still need to forward the port to the worker.
Updated by mkittler about 6 years ago
Since the XML already contains creating a serial console and also the guest already has it enabled by default (can I rely on that for SUSE?) it seems the missing part is making it available to the test. So I need to implement ssh $VIRSH_USERNAME@$VIRSH_HOSTNAME -t 'screen $(virsh ttyconsole openQA-SUT-$VIRSH_INSTANCE)'
somewhere in the backend. Likely I can just use backend::baseclass::new_ssh_connection
for that.
Updated by mkittler about 6 years ago
Instead of using screen and passing the pts
device manually, I can also just use virsh console openQA-SUT-1
. But that leads so an interesting error message:
Connected to domain openQA-SUT-1
Escape character is ^]
error: Operation schlug fehl: Aktive Konsol-Sitzung existiert für diese Domain
Searching in our code, it turns out that we actually already establish a serial connection in backend::svirt::start_serial_grab
. So it seems I only need to expose that $chan
object which is created there as a console to the test API?
Updated by mkittler about 6 years ago
- Status changed from In Progress to Feedback
I mark it as feedback before going in the wrong direction again.
Updated by mkittler about 6 years ago
The serial console connection which is already in place is used for the serial log. Connecting to this serial console via virsh console
fails with the error from my previous comment. Connecting with screen
works but it is unusable because characters are distributed randomly to only one of the 'clients'.
If sharing the existing serial console is not an option, one would assume that adding an additional serial console to the XML/libvirt config and use that instead would be a solution. However, doing this
+sub add_serial_console {
+ my ($self, $args) = @_;
+
+ my $doc = $self->{domainxml};
+ my $devices = $self->{devices_element};
+ my $port = $args->{port} // '0';
+
+ my $console = $doc->createElement('console');
+ $console->setAttribute(type => 'pty');
+ my $target = $doc->createElement('target');
+ $target->setAttribute(port => $port);
+ $console->appendChild($target);
+ $devices->appendChild($console);
+
+ return;
+}
+
leads to the error:
[2018-10-26T11:00:45.0709 CEST] [debug] Command executed: virsh define /var/lib/libvirt/images/openQA-SUT-1.xml
[2018-10-26T11:00:46.0006 CEST] [debug] Command's stderr:
error: Failed to define domain from /var/lib/libvirt/images/openQA-SUT-1.xml
error: Nicht unterstützte Konfiguration: Nur die erste Konsole kann ein serieller Port sein
[2018-10-26T11:00:46.0112 CEST] [debug] {
'console' => 'svirt',
'json_cmd_token' => 'ldZSLIEp',
'function' => 'define_and_start',
'cmd' => 'backend_proxy_console_call',
'args' => []
}
virsh define failed at /hdd/openqa-devel/repos/os-autoinst/consoles/sshVirtsh.pm line 493.
So creating an additional serial console isn't possible - at least not that way. Maybe I can configure it instead like this: https://unix.stackexchange.com/questions/383460/multiple-virsh-kvm-guest-consoles-without-graphics#answers-header
Updated by mkittler about 6 years ago
- Status changed from Feedback to In Progress
It isn't possible to add a 2nd serial console, but it is possible to open another serial port. So I'll just do that:
<serial type="pty">
<target port="1"/>
</serial>
In my tests it seems to work:
- in SUT:
sudo systemctl start serial-getty@ttyS1
- on host
sudo virsh dumpxml openQA-SUT-1
to get the path of the device, eg./dev/pts/21
sudo screen /dev/pts/21
Maybe there's another virsh
command to simplify the procedure on the host? Putting source
tags in the XML config for fix paths would also be an option but it hasn't had any effect in my tests.
Updated by mkittler about 6 years ago
possibly conflicting PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6073
Updated by pvorel about 6 years ago
That's my pull request. It's not conflicting :).
If you manage to have some console, we just update select_serial_terminal()
.
Updated by mkittler about 6 years ago
That was more a note for myself. I mean not conflicting in the sense Git will complain - more in the way like I'll have to take that new abstraction into account.
Updated by mkittler about 6 years ago
WIP PR: https://github.com/os-autoinst/os-autoinst/pull/1048
The configuration of the virtual machine to create the serial console seems to work.
Making use of it is still WIP:
- So far I attempt to read the path of the serial device exposed by the virtual machine using a Perl script which parses the libvirtd XML. That works because the XML used to define the domain initially becomes modified when the virtual machine is running to contain that information. So far I was unable to just set a fixed path for the device in the initial XML config.
- Then I try to connect via screen to it.
- 1. and 2. done via SSH on the svirt host. So I try to expose that SSH channel as console which can be added/selected via
$self->add_console('sut-serial', 'ssh-virtsh-sut', {});
/select_console('sut-serial');
.
Updated by mkittler about 6 years ago
- File screenshot_20181106_171219.png screenshot_20181106_171219.png added
- File screenshot_20181106_171235.png screenshot_20181106_171235.png added
See my latest PR comment: https://github.com/os-autoinst/os-autoinst/pull/1048#issuecomment-436237366
I did the required validation for Xen and Hyper-V. My changes at least didn't beak anything. I also added some documentation about the svirt backend.
So what's left are unit tests and likely some fine tuning or adjustments according feedback.
Updated by pvorel about 6 years ago
@mkittler Thank you for the implementation.
Can you please post a link to openQA where these tests are, so we can have a look more?
Can you please clone on your machine install_ltp job on svirt for s390x?
https://openqa.suse.de/tests/2240108
Or any other ltp_* job from kernel group?
https://openqa.suse.de/tests/overview?distri=sle&version=12-SP4&build=0455&groupid=155
Updated by mkittler about 6 years ago
@pvorel
The tests are only on my local machine where I often switch databases and configuration (I'm mainly develop openQA itself) so sending a link is not of much use. But I've sent the links for the Xen and Hyper-V validation to @mnovak at the time they were valid and he said "that's fine with me".
But maybe you're just interested in how I changed the test distribution code for my testing? That's the diff: https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...Martchus:svirt3
I could clone the test. However, it would not magically use the new console. This could be improved in accordance with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6073.
Updated by pvorel about 6 years ago
The tests are only on my local machine where I often switch databases and configuration (I'm mainly develop openQA itself) so sending a link is not of much use. But I've sent the links for the Xen and Hyper-V validation to @mnovak at the time they were valid and he said "that's fine with me".
What a shame. I'd like to test your patch with LTP, but I have no svirt worker configured in my machine (http://quasar.suse.cz/). rpalethorpe also works on LTP, maybe he has tested it.
But maybe you're just interested in how I changed the test distribution code for my testing? That's the diff: https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...Martchus:svirt3
I cannot write my comments, as it's not a PR. How about create PR and add [WIP]? Maybe too early stage.
Just some notes:
boot_to_desktop.pm: password is in $testapi::password, can you use it. Is it always the same or would it be handy to allow to set it with variable?
I could clone the test. However, it would not magically use the new console. This could be improved in accordance with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6073.
Yes, that's my PR. Like trinity.pm also this test uses select_virtio_console() (in install_ltp.pm) + many other our kernel tests. So the goal is to have your changes integrated into select_serial_terminal(), once my PR is merged.
Updated by mkittler about 6 years ago
How about create PR and add [WIP]? Maybe too early stage.
These changes are really just for my own testing.
it always the same or would it be handy to allow to set it with variable?
It is just the usual login password. I've just quickly hacked that, of course it makes sense to use $testapi::password
here.
So the goal is to have your changes integrated into select_serial_terminal(), once my PR is merged.
Makes sense.
Updated by mkittler about 6 years ago
- Status changed from In Progress to Feedback
So right now I'm waiting for https://github.com/os-autoinst/os-autoinst/pull/1048 to be merged so I can get feedback.
Note that the additional serial console is right now a bit hard to use in practice. That's because one has to execute type_string("sudo systemctl start serial-getty\@ttyS1\n");
at some point after the SUT has been started. I'm not sure where the best place in the test distribution for executing that line would be.
It would also make sense to execute type_string("sudo systemctl enable --now serial-getty\@ttyS1\n");
after the installation to make the serial console available after rebooting without extra effort.
Updated by okurz about 6 years ago
That's a funny proposal because it sounds like exactly the opposite of what we wanted to do together with riafarov to avoid getty prompts confusing the expected output on the primary serial device. However I assume we are good as long as we are talking about a dedicated serial device for each: serial output matching on the primary device and a serial terminal on a secondary device, right?
Updated by mkittler about 6 years ago
This would be a dedicated serial device (which is the whole reason for the extra effort I mentioned in my previous comment).
Btw, maybe it is also possible use additional boot options (like bootmenu_type_console_params
in bootloader_setup.pm
) to initialize the additional tty console (instead of using systemd commands).
Updated by okurz about 6 years ago
PR is merged and already deployed on o3 at least, didn't check for osd. However, we should ensure that we also actually use this feature in tests. Should we do that in this ticket or a follow-up?
Updated by coolo about 6 years ago
- Assignee changed from mkittler to pvorel
- Target version deleted (
Current Sprint)
Petr, please take over the verification/usage
Updated by okurz almost 6 years ago
- Related to coordination #44843: [qe-core][functional][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module) added
Updated by pvorel almost 6 years ago
- Related to action #45863: [s390x][svirt][ltp] Fix serial terminal console implementation for svirt backend and use it's output added
Updated by pvorel almost 6 years ago
I created for work, which is still needed to be done separate ticket #45863.
Updated by pvorel almost 6 years ago
- Status changed from Feedback to Resolved
- Assignee deleted (
pvorel)