action #36457

Implement serial terminal console for svirt backend

Added by pvorel almost 2 years ago. Updated about 1 year ago.

Status:ResolvedStart date:23/05/2018
Priority:HighDue date:
Assignee:-% Done:

0%

Category:Feature requests
Target version:-
Difficulty:hard
Duration:

Description

Probably not easy to implement, but kernel & networking (and other console tests) would benefit from it:

  • Being much faster
  • Having text based console logs (instead of screenshots of text)

screenshot_20181106_171219.png (65.5 KB) mkittler, 06/11/2018 04:12 pm

screenshot_20181106_171235.png - failed uploading logs because of my firewall but passed critical sections (66 KB) mkittler, 06/11/2018 04:12 pm

7112
7115

Related issues

Related to openQA Project - action #32599: [tools][openqa][ppc64le][s390x_zkvm] Kernel log missing f... Feedback 01/03/2018
Related to openQA Tests - action #44843: [functional][u][epic] Cleanup the use of serial-/virtio-/... Blocked 13/12/2018
Related to openQA Project - action #45863: [s390x][svirt][ltp] Fix serial terminal console implement... Resolved
Duplicated by openQA Project - action #41840: [sle][s390x][s390-kvm][tools][ltp] Enable virtio_terminal... Rejected 01/10/2018

History

#1 Updated by pvorel almost 2 years ago

  • Related to action #32599: [tools][openqa][ppc64le][s390x_zkvm] Kernel log missing from serial0.txt added

#2 Updated by coolo almost 2 years ago

To begin with: there is no zkvm backend - and it's not ppc64le related. And I thought you're heading for hardware tests anyway.

#3 Updated by pvorel almost 2 years ago

  • Subject changed from [tools][openqa][ppc64le][s390x_zkvm] Implement virtio console for zkvm backend to [tools][openqa][s390x_zkvm] Implement virtio console for svirt backend

#4 Updated by pvorel almost 2 years ago

@coolo: Thanks for pointing out errors.

#5 Updated by coolo over 1 year ago

  • Duplicated by action #41840: [sle][s390x][s390-kvm][tools][ltp] Enable virtio_terminal console (VIRTIO_CONSOLE) on svirt backend added

#6 Updated by coolo over 1 year ago

  • Subject changed from [tools][openqa][s390x_zkvm] Implement virtio console for svirt backend to Implement virtio console for svirt backend
  • Category set to 132
  • Priority changed from Normal to High
  • Target version set to Ready

#7 Updated by mkittler over 1 year ago

  • Assignee set to mkittler

So far I'm just looking into how I can setup a development environment for the svirt backend. Despite the initial topic of the ticket, the svirt backend looks to me architecture neutral. Maybe I could even just ssh to my own machine where I have virsh and qemu installed. For me it would be easier not having to deal with s390pb (or whatever I could use for the jump-host?).

When I'm really starting to work on this I'm going to set the status to in progress.

#8 Updated by coolo over 1 year ago

Yes, you can just use localhost as svirt host

#9 Updated by coolo over 1 year ago

  • Target version changed from Ready to Current Sprint

#10 Updated by mkittler over 1 year ago

  • Status changed from New to In Progress

I managed to get jobs running with the svirt backend locally using a worker config like I documented here. If you like I can also polish and upstream this documentation.

I also managed to enable virtio in qemu when started via libvirtd. To test this I adjusted the test distribution.

Right now it can not connect to the virtio console because there are issues getting the path of the socket right:

[2018-10-25T15:47:40.0284 CEST] [debug] <<< consoles::virtio_terminal::open_socket(socket_path='/hdd/openqa-devel/openqa/pool/2/virtio_console')
DIE Can't connect($fh, '�/hdd/openqa-devel/openqa/pool/2/virtio_console��������������������������������������������������������������'): Connection refused at /hdd/openqa-devel/repos/os-autoinst/consoles/virtio_terminal.pm line 127

 at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 80.
    backend::baseclass::die_handler(autodie::exception=HASH(0x5563a74c7270)) called at (eval 178) line 33
    consoles::virtio_terminal::__ANON__(GLOB(0x5563a74d4f70), "\x{1}\x{0}/hdd/openqa-devel/openqa/pool/2/virtio_console\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}"...) called at /hdd/openqa-devel/repos/os-autoinst/consoles/virtio_terminal.pm line 127
    consoles::virtio_terminal::open_socket(consoles::virtio_terminal=HASH(0x5563a6fa4e38)) called at /hdd/openqa-devel/repos/os-autoinst/consoles/virtio_terminal.pm line 136
    consoles::virtio_terminal::activate(consoles::virtio_terminal=HASH(0x5563a6fa4e38)) called at /hdd/openqa-devel/repos/os-autoinst/consoles/console.pm line 85
    consoles::console::select(consoles::virtio_terminal=HASH(0x5563a6fa4e38)) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 529
    backend::baseclass::select_console(backend::svirt=HASH(0x5563a7bc1ae8), HASH(0x5563a7508d38)) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 75
    backend::baseclass::handle_command(backend::svirt=HASH(0x5563a7bc1ae8), HASH(0x5563a74d1370)) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 487
    backend::baseclass::check_socket(backend::svirt=HASH(0x5563a7bc1ae8), IO::Handle=GLOB(0x5563a66a96a8), 0) called at /hdd/openqa-devel/repos/os-autoinst/backend/svirt.pm line 236
    backend::svirt::check_socket(backend::svirt=HASH(0x5563a7bc1ae8), IO::Handle=GLOB(0x5563a66a96a8), 0) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 246
    eval {...} called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 156
    backend::baseclass::run_capture_loop(backend::svirt=HASH(0x5563a7bc1ae8)) called at /hdd/openqa-devel/repos/os-autoinst/backend/baseclass.pm line 129
    backend::baseclass::run(backend::svirt=HASH(0x5563a7bc1ae8), 13, 16) called at /hdd/openqa-devel/repos/os-autoinst/backend/driver.pm line 92
    backend::driver::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x5563a785b458)) called at /usr/lib/perl5/vendor_perl/5.26.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
    eval {...} called at /usr/lib/perl5/vendor_perl/5.26.2/Mojo/IOLoop/ReadWriteProcess.pm line 325
    Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x5563a785b458), CODE(0x5563a776fc40)) called at /usr/lib/perl5/vendor_perl/5.26.2/Mojo/IOLoop/ReadWriteProcess.pm line 476
    Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x5563a785b458)) called at /hdd/openqa-devel/repos/os-autoinst/backend/driver.pm line 94
    backend::driver::start(backend::driver=HASH(0x5563a676e320)) called at /hdd/openqa-devel/repos/os-autoinst/backend/driver.pm line 50
    backend::driver::new("backend::driver", "svirt") called at /hdd/openqa-devel/repos/os-autoinst/isotovideo line 184
    main::init_backend() called at /hdd/openqa-devel/repos/os-autoinst/isotovideo line 249

However, I can manually connect to the socket. To do that I've just paused the test using the developer mode before it would attempt to connect and executed the Perl code myself:

sudo perl -I. -de1
...
  DB<1> use consoles::virtio_terminal;
  DB<2>  $term = consoles::virtio_terminal->new();
  DB<3> print($term->socket_path);
/hdd/openqa-devel/repos/os-autoinst/virtio_console
  DB<4> $term->open_socket();
[2018-10-25T15:19:03.0360 CEST] [debug] <<< consoles::virtio_terminal::open_socket(socket_path='/hdd/openqa-devel/repos/os-autoinst/virtio_console')
Could not find /hdd/openqa-devel/repos/os-autoinst/virtio_console at (eval 108)[/usr/lib/perl5/5.26.2/perl5db.pl:738] line 2.
  DB<5> $term->{socket_path} = '/var/lib/libvirt/qemu/channel/target/domain-4-openQA-SUT-1/org.openqa.console.virtio_console';
  DB<6>  $term->open_socket();
[2018-10-25T15:19:09.0031 CEST] [debug] <<< consoles::virtio_terminal::open_socket(socket_path='/var/lib/libvirt/qemu/channel/target/domain-4-openQA-SUT-1/org.openqa.console.virtio_console')
<no error>

So there are multiple things TODO:

  1. Tell libvirt to create the socket at a specific location or try to detect it at its default location.
  2. Make the virtio socket on the svirt host available on the worker where the test is executed (eg. SSH tunnel?).

#11 Updated by coolo over 1 year ago

  • Subject changed from Implement virtio console for svirt backend to Implement serial terminal console for svirt backend

Forget about qemu and it's virtio - I thought I made myself clear there. They want this for all backends

#12 Updated by mkittler over 1 year ago

Ok, so I guess I'll have to do it like described here: https://help.ubuntu.com/community/KVM/Access

So the modification to the XML will become even easier but I will need to configure the guest. And I still need to forward the port to the worker.

#13 Updated by mkittler over 1 year ago

Since the XML already contains creating a serial console and also the guest already has it enabled by default (can I rely on that for SUSE?) it seems the missing part is making it available to the test. So I need to implement ssh $VIRSH_USERNAME@$VIRSH_HOSTNAME -t 'screen $(virsh ttyconsole openQA-SUT-$VIRSH_INSTANCE)' somewhere in the backend. Likely I can just use backend::baseclass::new_ssh_connection for that.

#14 Updated by mkittler over 1 year ago

Instead of using screen and passing the pts device manually, I can also just use virsh console openQA-SUT-1. But that leads so an interesting error message:

Connected to domain openQA-SUT-1
Escape character is ^]
error: Operation schlug fehl: Aktive Konsol-Sitzung existiert für diese Domain

Searching in our code, it turns out that we actually already establish a serial connection in backend::svirt::start_serial_grab. So it seems I only need to expose that $chan object which is created there as a console to the test API?

#15 Updated by mkittler over 1 year ago

  • Status changed from In Progress to Feedback

I mark it as feedback before going in the wrong direction again.

#16 Updated by mkittler over 1 year ago

The serial console connection which is already in place is used for the serial log. Connecting to this serial console via virsh console fails with the error from my previous comment. Connecting with screen works but it is unusable because characters are distributed randomly to only one of the 'clients'.

If sharing the existing serial console is not an option, one would assume that adding an additional serial console to the XML/libvirt config and use that instead would be a solution. However, doing this

+sub add_serial_console {
+    my ($self, $args) = @_;
+
+    my $doc     = $self->{domainxml};
+    my $devices = $self->{devices_element};
+    my $port = $args->{port} // '0';
+
+    my $console = $doc->createElement('console');
+    $console->setAttribute(type        => 'pty');
+    my $target = $doc->createElement('target');
+    $target->setAttribute(port => $port);
+    $console->appendChild($target);
+    $devices->appendChild($console);
+
+    return;
+}
+

leads to the error:

[2018-10-26T11:00:45.0709 CEST] [debug] Command executed: virsh  define /var/lib/libvirt/images/openQA-SUT-1.xml
[2018-10-26T11:00:46.0006 CEST] [debug] Command's stderr:
error: Failed to define domain from /var/lib/libvirt/images/openQA-SUT-1.xml
error: Nicht unterstützte Konfiguration: Nur die erste Konsole kann ein serieller Port sein
[2018-10-26T11:00:46.0112 CEST] [debug] {
  'console' => 'svirt',
  'json_cmd_token' => 'ldZSLIEp',
  'function' => 'define_and_start',
  'cmd' => 'backend_proxy_console_call',
  'args' => []
}
virsh define failed at /hdd/openqa-devel/repos/os-autoinst/consoles/sshVirtsh.pm line 493.

So creating an additional serial console isn't possible - at least not that way. Maybe I can configure it instead like this: https://unix.stackexchange.com/questions/383460/multiple-virsh-kvm-guest-consoles-without-graphics#answers-header

#17 Updated by mkittler over 1 year ago

  • Status changed from Feedback to In Progress

It isn't possible to add a 2nd serial console, but it is possible to open another serial port. So I'll just do that:

<serial type="pty">
<target port="1"/>
</serial>

In my tests it seems to work:

  • in SUT: sudo systemctl start serial-getty@ttyS1
  • on host
    • sudo virsh dumpxml openQA-SUT-1 to get the path of the device, eg. /dev/pts/21
    • sudo screen /dev/pts/21

Maybe there's another virsh command to simplify the procedure on the host? Putting source tags in the XML config for fix paths would also be an option but it hasn't had any effect in my tests.

#19 Updated by pvorel over 1 year ago

That's my pull request. It's not conflicting :).
If you manage to have some console, we just update select_serial_terminal().

#20 Updated by mkittler over 1 year ago

That was more a note for myself. I mean not conflicting in the sense Git will complain - more in the way like I'll have to take that new abstraction into account.

#21 Updated by mkittler over 1 year ago

WIP PR: https://github.com/os-autoinst/os-autoinst/pull/1048

The configuration of the virtual machine to create the serial console seems to work.

Making use of it is still WIP:

  1. So far I attempt to read the path of the serial device exposed by the virtual machine using a Perl script which parses the libvirtd XML. That works because the XML used to define the domain initially becomes modified when the virtual machine is running to contain that information. So far I was unable to just set a fixed path for the device in the initial XML config.
  2. Then I try to connect via screen to it.
  3. 1. and 2. done via SSH on the svirt host. So I try to expose that SSH channel as console which can be added/selected via $self->add_console('sut-serial', 'ssh-virtsh-sut', {});/select_console('sut-serial');.

#22 Updated by mkittler over 1 year ago

See my latest PR comment: https://github.com/os-autoinst/os-autoinst/pull/1048#issuecomment-436237366

I did the required validation for Xen and Hyper-V. My changes at least didn't beak anything. I also added some documentation about the svirt backend.

So what's left are unit tests and likely some fine tuning or adjustments according feedback.

#23 Updated by pvorel over 1 year ago

@mkittler Thank you for the implementation.

Can you please post a link to openQA where these tests are, so we can have a look more?

Can you please clone on your machine install_ltp job on svirt for s390x?
https://openqa.suse.de/tests/2240108

Or any other ltp_* job from kernel group?
https://openqa.suse.de/tests/overview?distri=sle&version=12-SP4&build=0455&groupid=155

#24 Updated by mkittler over 1 year ago

@pvorel
The tests are only on my local machine where I often switch databases and configuration (I'm mainly develop openQA itself) so sending a link is not of much use. But I've sent the links for the Xen and Hyper-V validation to @mnovak at the time they were valid and he said "that's fine with me".

But maybe you're just interested in how I changed the test distribution code for my testing? That's the diff: https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...Martchus:svirt3

I could clone the test. However, it would not magically use the new console. This could be improved in accordance with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6073.

#25 Updated by pvorel over 1 year ago

The tests are only on my local machine where I often switch databases and configuration (I'm mainly develop openQA itself) so sending a link is not of much use. But I've sent the links for the Xen and Hyper-V validation to @mnovak at the time they were valid and he said "that's fine with me".

What a shame. I'd like to test your patch with LTP, but I have no svirt worker configured in my machine (http://quasar.suse.cz/). rpalethorpe also works on LTP, maybe he has tested it.

But maybe you're just interested in how I changed the test distribution code for my testing? That's the diff: https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...Martchus:svirt3

I cannot write my comments, as it's not a PR. How about create PR and add [WIP]? Maybe too early stage.
Just some notes:
boot_to_desktop.pm: password is in $testapi::password, can you use it. Is it always the same or would it be handy to allow to set it with variable?

I could clone the test. However, it would not magically use the new console. This could be improved in accordance with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6073.

Yes, that's my PR. Like trinity.pm also this test uses select_virtio_console() (in install_ltp.pm) + many other our kernel tests. So the goal is to have your changes integrated into select_serial_terminal(), once my PR is merged.

#26 Updated by mkittler over 1 year ago

How about create PR and add [WIP]? Maybe too early stage.

These changes are really just for my own testing.

it always the same or would it be handy to allow to set it with variable?

It is just the usual login password. I've just quickly hacked that, of course it makes sense to use $testapi::password here.

So the goal is to have your changes integrated into select_serial_terminal(), once my PR is merged.

Makes sense.

#27 Updated by mkittler over 1 year ago

  • Status changed from In Progress to Feedback

So right now I'm waiting for https://github.com/os-autoinst/os-autoinst/pull/1048 to be merged so I can get feedback.

Note that the additional serial console is right now a bit hard to use in practice. That's because one has to execute type_string("sudo systemctl start serial-getty\@ttyS1\n"); at some point after the SUT has been started. I'm not sure where the best place in the test distribution for executing that line would be.

It would also make sense to execute type_string("sudo systemctl enable --now serial-getty\@ttyS1\n"); after the installation to make the serial console available after rebooting without extra effort.

#28 Updated by okurz over 1 year ago

That's a funny proposal because it sounds like exactly the opposite of what we wanted to do together with riafarov to avoid getty prompts confusing the expected output on the primary serial device. However I assume we are good as long as we are talking about a dedicated serial device for each: serial output matching on the primary device and a serial terminal on a secondary device, right?

#29 Updated by mkittler over 1 year ago

This would be a dedicated serial device (which is the whole reason for the extra effort I mentioned in my previous comment).

Btw, maybe it is also possible use additional boot options (like bootmenu_type_console_params in bootloader_setup.pm) to initialize the additional tty console (instead of using systemd commands).

#30 Updated by okurz over 1 year ago

PR is merged and already deployed on o3 at least, didn't check for osd. However, we should ensure that we also actually use this feature in tests. Should we do that in this ticket or a follow-up?

#31 Updated by coolo over 1 year ago

  • Assignee changed from mkittler to pvorel
  • Target version deleted (Current Sprint)

Petr, please take over the verification/usage

#32 Updated by okurz over 1 year ago

  • Related to action #44843: [functional][u][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module) added

#33 Updated by pvorel about 1 year ago

  • Related to action #45863: [s390x][svirt][ltp] Fix serial terminal console implementation for svirt backend and use it's output added

#34 Updated by pvorel about 1 year ago

I created for work, which is still needed to be done separate ticket #45863.

#35 Updated by pvorel about 1 year ago

  • Status changed from Feedback to Resolved
  • Assignee deleted (pvorel)

Also available in: Atom PDF