action #68983
closedcoordination #36712: [saga] Use YaST specific framework for GUI testing
[y][timeboxed:24h] Explore possibilities for test written using libyui
Added by riafarov over 4 years ago. Updated about 4 years ago.
0%
Description
We have our framework pretty much in a good shape to be used for testing.
So we should design architecture for it and find a way to integrate it with openQA.
Requirements for the solution:
- Integration in openQA
- Ease of running locally, e.g. by dev
- Ease of maintenance for multiple versions of distributions
* Clear structure for the test development
Updated by riafarov over 4 years ago
- Subject changed from [y] Explore possibilities for test written using libyui to [y][timeboxed:24h] Explore possibilities for test written using libyui
- Description updated (diff)
- Status changed from New to Workable
Updated by riafarov over 4 years ago
We should keep in mind not only yast modules, but also installation tests. This means that we might use multimachine tests, but there we have limitations there (line no MM on s390x). For the solution to run code on the worker directly, we also need to discuss it with the tools team as we will need ruby dependencies there. Docker container might be a solution to that too.
Also, https://github.com/libyui/libyui-rest-api/blob/master/README.md#reuse-of-the-socket can help to spawn server side quickly.
We also explore integration xUnit.
Updated by JERiveraMoya over 4 years ago
- Status changed from Workable to Feedback
Options where to setup ruby stack:
Multi-machine
There are two things to consider:
Dependencies:
According to openQA developers, from openQA perspective should not be problem to do things like PARALLEL_WITH=A@, in other words, to have parallel jobs
with different architectures (we are not sure if this is really working right now). Anton mentioned that FLAVOR is hardlinked to some certain ARCH so it is not possible
to set different values to same variable during single isos post call, but anyway those "details" should be adjustable if it isn't already working according to developer.
Regarding a possible architecture, to have one controller with the ruby stack and several jobs in parallel depending on it might not make sense, because the VMs needs to tell
the controller how to create the MVC (arch and other params) plus we would need to ssh each other. In MM tests that connection among the components should not happen, each machine should do their work even for retrieving logs, currently we have some hacks in openQA code to just type without waiting return code to do remote installation. Therefore for each arch/flavor we need two jobs which will not need to talk at all. That means that if we plan to add to our ruby client the possibility to open via ssh the yast module in mode server that would be typed in a non-safe way, so would need to open the modules in the VM itself before sending the rest calls and have in mind implement a different way to open the module when running without openQA.Networking:
The main problem with multi-machine is the networking setup.
Currently there is not support for ppc64le and s390x.
According to Mathias to have it in s390x KVM it would need to happen codewise on the LPAR (s390p8 for example) and the XML builder, which would require some
Network expertise. Checking the code in bootloader_zkvm, basically the backend implemented for svirt allows us to get a console object which provide certain actions, one of them is to edit XML required to run the guests, so we would need to add a custom networking there.
For other flavor of s390x or ppc64le we have to figure it out and might be not so straightforward.
Worker
There are two things to consider here:
- Lack of Backend: To have the ruby packages available in the worker or even to have a container with the ruby stack and our client seems not to be the issue, and we could use some help from Sergio about how to salt it. The main problem is that we will be sending commands without any control mechanism due to there is not backend between the worker and the guest.
Nevertheless I tried a few things to see what I could find:
In our ruby project currently we don't use any gem that could require some extra zypper, so we just zypper in ruby2.5-rubygem-bundler to get bundler to run the rspec.
I was able to check that ruby version in one of the worker is ruby 2.5.8p224 (2020-03-31 revision 67882) [x86_64-linux-gnu] and in case we don't update it in the worker to 2.7, the project still works with that one. Running commands directly on the worker with print qx{zypper in ruby2.5-rubygem-bundler} I faced the issue of permission because those are running in pool folder in the worker:
output of id: uid=486(_openqa-worker) gid=65534(nogroup) groups=65534(nogroup),483(kvm)
output of pwd: /var/lib/openqa/pool/19
Example of output: https://openqa.suse.de/tests/4466521/file/autoinst-log.txt
- Lack of synchronization: We would need to implement locks so the guest could know where to finish any task.
Externally
It suffers of the same issues than the worker, lack of backend and synchronization.
Synchronization with VMs created in openqa, all of them spawn by isos post need to check-in somehow with this external component and in turn this component would send back (in parallel?) to the right ip and port the corresponding rest calls produced by underlying MVC code depending of the architecture. There are several places to run it (with not cost more than electricity):
- gitlab CI runners (I didn't explore it)
- QA internal k8s cluster (I don't know where it is but exists)
- http://cloud.suse.de: I checked with Santi that there is connectivity, but introduce a dependency outside openqa, for example ldap was broken in cloud when trying to test out something. I was able to login in the end.
Local
It would mess up with the SUT, basically changing what we test.
Ease of running locally
Even if we wrap the solution with something else, it would still be easy to run (even in ruby 2.5).
Ease of maintenance for multiple versions of distris
This is part of the MVC to set a particular set of rest calls (or even a particular way to open the module if there is any difference and this is included as part of the MVC).
If we use some external machine, we need to provide the required info for the MVC.
Integration of xUnit
Using xunit and api function http://open.qa/api/testapi/#_parse_extra_log I found a couple of test modules:
run_ltp: https://openqa.suse.de/tests/4342832
slurm_master: https://openqa.suse.de/tests/4342980
Regarding how to create xunit from rspec we were checking in previous task this plugin: https://progress.opensuse.org/issues/54305
Summary¶
The options to run it on the worker or in some external service/machine/cloud instance (regardless if they have a container or not) would require to create a new Backend, otherwise we would need to run things in a hacky way and regarding synchronization unless we can find a different flow that does not require such communication (for instance, have the guests outside too, but we would need to boot the image by our own without openQA), we would be reinventing the wheel.
Local only fits for an installed system to test YaST modules, but not for the installer itself.
Work on networking setup in MM if we succeed would bring two benefits, to have MM in another architecture and to be able to run our ruby stack using a proper Backend and basic Synchronization.
Updated by JERiveraMoya over 4 years ago
Previous investigation was based on the assumption to run only commands in the worker, but we can mix both, commands that reach the worker and command that use the backend and reach the SUT in a controlled way. I dive in a bit more and this is what I found:
We can have a test module file which install packages in the VM to be able to start YaST modules in server mode. For example, in SLE would need:
select_console 'root-console';
zypper_call("ar https://download.opensuse.org/repositories/YaST:/Head/openSUSE_Leap_15.2/YaST:Head.repo");
zypper_call("--gpg-auto-import-keys ref");
zypper_call("in libyui-rest-api");
# we should have the package available for the distribution and not need to add a repo,
# because without change vendor for all the packages (which doesn't make sense because it is modifying the SUT)
# it should work just installing one package, but not in this example.
zypper_call("up --allow-vendor-change -r YaST_Head");
Additional things that can be done:
my $ip = script_output("ip -4 addr show eth0 | grep -oP '(?<=inet\\s)\\d+(\\.\\d+){3}'"); # get IP
systemctl 'stop firewalld'; # stop firewall
Once setup, we can launch the module with all params we need:
select_console 'x11';
$self->launch_yast2_module_x11('host', match_timeout => 30);
Until this point visually this would look like this: https://openqa.suse.de/tests/4475229#step/yast2_hostnames/28
Now we can run some code in the worker (the test file didn't finish so the VM is not destroyed yet!) which will give us some inside about where we are running things:
print qx{ip a}; # list of network interfaces
print qx{ping -I br1 -c 4 $ip}; # ping VM ip (it doesn't make sense as I will explain later)
print qx{id}; # user
print qx{cat /etc/os-release}; # worker distri
print qx{ruby --version}; # ruby version
print qx{sudo lsof -i -P -n | grep LISTEN}; # Check open ports
Afterwards we can clone the repo branch that I used for this PoC and zypper the package manager for ruby
print qx{git clone -b poc_ruby_integration --single-branch https://github.com/jknphy/libyui_client.git};
print qx{sudo zypper --non-interactive in ruby2.5-rubygem-bundler}; # be careful with that command, it is possible to run sudo without being asked for password!(it is salted that way)
Be aware of that running command separately we cannot change directory in one and assume we are in a different directory in another one, so it should run as a whole in that case, just check any time along the way:
print qx{echo "\$PWD"};
Bundle needs writable home directory and in the work it is not, so we can add variable env to run it and check the content of the directory:
print qx{HOME=\$PWD bundle install --path \$PWD/libyui_client/.vendor/bundler --gemfile \$PWD/libyui_client/Gemfile};
print qx{ls -lah \$PWD/libyui_client};
We can run our rspec without any problem:
print qx{cd libyui_client ; HOME=\$PWD bundle exec rspec};
And the only final step is to run some sample.rb that interact with the UI, for example:
require 'libyui_client'
# app = LibyuiClient::App.new(host: ENV['GUEST_IP'] , port: '9999')
app = LibyuiClient::App.new(host: 'localhost' , port: '9998')
button = app.button(id: 'add')
button.click
And this is the command in the worker to run it:
print qx{cd libyui_client ; HOME=\$PWD bundle exec ruby lib/libyui_client/sample.rb};
This final step does not work because the is connection directly to the VM as the setup that we use in openQA is the default one, which allows the guest to access networking resources from the host
but not in the other direction. I explore this port forwarding in my local env and works nicely, running the ruby client in localhost with port 6666 and checking that in the guest we can click the button with the module open in port 9999.
qemu-system-x86_64 -m 1024 -cpu qemu64 -enable-kvm \
-hda image_test.qcow2 \
-bios /usr/share/qemu/ovmf-x86_64.bin \
-netdev user,id=qanet0**,hostfwd=tcp::6666-:9999** -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56
But at the moment openQA cannot allow to overwrite the netdev, only append parameters, which I also try, but that forwarding can only be defined when the netdev is defined.
Here are some logs of the whole thing:
https://openqa.suse.de/tests/4478129/file/autoinst-log.txt