openQA Project (public)

To begin with: there is no zkvm backend - and it's not ppc64le related. And I thought you're heading for hardware tests anyway.

@coolo: Thanks for pointing out errors.

Yes, you can just use localhost as svirt host

Ok, so I guess I'll have to do it like described here: https://help.ubuntu.com/community/KVM/Access

So the modification to the XML will become even easier but I will need to configure the guest. And I still need to forward the port to the worker.

Since the XML already contains creating a serial console and also the guest already has it enabled by default (can I rely on that for SUSE?) it seems the missing part is making it available to the test. So I need to implement ssh $VIRSH_USERNAME@$VIRSH_HOSTNAME -t 'screen $(virsh ttyconsole openQA-SUT-$VIRSH_INSTANCE)' somewhere in the backend. Likely I can just use backend::baseclass::new_ssh_connection for that.

Instead of using screen and passing the pts device manually, I can also just use virsh console openQA-SUT-1. But that leads so an interesting error message:

Connected to domain openQA-SUT-1
Escape character is ^]
error: Operation schlug fehl: Aktive Konsol-Sitzung existiert für diese Domain

Searching in our code, it turns out that we actually already establish a serial connection in backend::svirt::start_serial_grab. So it seems I only need to expose that $chan object which is created there as a console to the test API?

The serial console connection which is already in place is used for the serial log. Connecting to this serial console via virsh console fails with the error from my previous comment. Connecting with screen works but it is unusable because characters are distributed randomly to only one of the 'clients'.

If sharing the existing serial console is not an option, one would assume that adding an additional serial console to the XML/libvirt config and use that instead would be a solution. However, doing this

+sub add_serial_console {
+    my ($self, $args) = @_;
+
+    my $doc     = $self->{domainxml};
+    my $devices = $self->{devices_element};
+    my $port = $args->{port} // '0';
+
+    my $console = $doc->createElement('console');
+    $console->setAttribute(type        => 'pty');
+    my $target = $doc->createElement('target');
+    $target->setAttribute(port => $port);
+    $console->appendChild($target);
+    $devices->appendChild($console);
+
+    return;
+}
+

leads to the error:

[2018-10-26T11:00:45.0709 CEST] [debug] Command executed: virsh  define /var/lib/libvirt/images/openQA-SUT-1.xml
[2018-10-26T11:00:46.0006 CEST] [debug] Command's stderr:
error: Failed to define domain from /var/lib/libvirt/images/openQA-SUT-1.xml
error: Nicht unterstützte Konfiguration: Nur die erste Konsole kann ein serieller Port sein
[2018-10-26T11:00:46.0112 CEST] [debug] {
  'console' => 'svirt',
  'json_cmd_token' => 'ldZSLIEp',
  'function' => 'define_and_start',
  'cmd' => 'backend_proxy_console_call',
  'args' => []
}
virsh define failed at /hdd/openqa-devel/repos/os-autoinst/consoles/sshVirtsh.pm line 493.

So creating an additional serial console isn't possible - at least not that way. Maybe I can configure it instead like this: https://unix.stackexchange.com/questions/383460/multiple-virsh-kvm-guest-consoles-without-graphics#answers-header

possibly conflicting PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6073

That's my pull request. It's not conflicting :).
If you manage to have some console, we just update select_serial_terminal().

That was more a note for myself. I mean not conflicting in the sense Git will complain - more in the way like I'll have to take that new abstraction into account.

WIP PR: https://github.com/os-autoinst/os-autoinst/pull/1048

The configuration of the virtual machine to create the serial console seems to work.

Making use of it is still WIP:

1. So far I attempt to read the path of the serial device exposed by the virtual machine using a Perl script which parses the libvirtd XML. That works because the XML used to define the domain initially becomes modified when the virtual machine is running to contain that information. So far I was unable to just set a fixed path for the device in the initial XML config.
2. Then I try to connect via screen to it.
3. 1. and 2. done via SSH on the svirt host. So I try to expose that SSH channel as console which can be added/selected via $self->add_console('sut-serial', 'ssh-virtsh-sut', {});/select_console('sut-serial');.

@mkittler Thank you for the implementation.

Can you please post a link to openQA where these tests are, so we can have a look more?

Can you please clone on your machine install_ltp job on svirt for s390x?
https://openqa.suse.de/tests/2240108

Or any other ltp_* job from kernel group?
https://openqa.suse.de/tests/overview?distri=sle&version=12-SP4&build=0455&groupid=155

@pvorel
The tests are only on my local machine where I often switch databases and configuration (I'm mainly develop openQA itself) so sending a link is not of much use. But I've sent the links for the Xen and Hyper-V validation to @mnovak at the time they were valid and he said "that's fine with me".

But maybe you're just interested in how I changed the test distribution code for my testing? That's the diff: https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...Martchus:svirt3

I could clone the test. However, it would not magically use the new console. This could be improved in accordance with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6073.

The tests are only on my local machine where I often switch databases and configuration (I'm mainly develop openQA itself) so sending a link is not of much use. But I've sent the links for the Xen and Hyper-V validation to @mnovak at the time they were valid and he said "that's fine with me".

What a shame. I'd like to test your patch with LTP, but I have no svirt worker configured in my machine (http://quasar.suse.cz/). rpalethorpe also works on LTP, maybe he has tested it.

But maybe you're just interested in how I changed the test distribution code for my testing? That's the diff: https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...Martchus:svirt3

I cannot write my comments, as it's not a PR. How about create PR and add [WIP]? Maybe too early stage.
Just some notes:
boot_to_desktop.pm: password is in $testapi::password, can you use it. Is it always the same or would it be handy to allow to set it with variable?

I could clone the test. However, it would not magically use the new console. This could be improved in accordance with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6073.

Yes, that's my PR. Like trinity.pm also this test uses select_virtio_console() (in install_ltp.pm) + many other our kernel tests. So the goal is to have your changes integrated into select_serial_terminal(), once my PR is merged.

How about create PR and add [WIP]? Maybe too early stage.

These changes are really just for my own testing.

it always the same or would it be handy to allow to set it with variable?

It is just the usual login password. I've just quickly hacked that, of course it makes sense to use $testapi::password here.

So the goal is to have your changes integrated into select_serial_terminal(), once my PR is merged.

Makes sense.

That's a funny proposal because it sounds like exactly the opposite of what we wanted to do together with riafarov to avoid getty prompts confusing the expected output on the primary serial device. However I assume we are good as long as we are talking about a dedicated serial device for each: serial output matching on the primary device and a serial terminal on a secondary device, right?

This would be a dedicated serial device (which is the whole reason for the extra effort I mentioned in my previous comment).

Btw, maybe it is also possible use additional boot options (like bootmenu_type_console_params in bootloader_setup.pm) to initialize the additional tty console (instead of using systemd commands).

PR is merged and already deployed on o3 at least, didn't check for osd. However, we should ensure that we also actually use this feature in tests. Should we do that in this ticket or a follow-up?

I created for work, which is still needed to be done separate ticket #45863.