QA » openQA Project » openQA Tests » qe-yam

Motivation¶

As per discussion on yast-devel mailing thread, it was decided that solution developed by Lada during the hackweek is a good starting point for the development.
It's also clear that we should not any efforts in bringing back macro player/recorder with YCP, but YCP bindings seem to stay for a longer period of time.

We need to get familiar with the changes, to decide how to proceed and estimate efforts, so this solution can be used on the production.

Here are useful links:

1. https://github.com/lslezak/cucumber-yast#installing-cucumber-ruby-gem
2. https://hackweek.suse.com/17/projects/yast-integration-tests-using-cucumber
3. https://github.com/libyui/libyui-qt/tree/http_server

Acceptance criteria¶

1. Identify current capabilities and how it can be used for YaST modules integration tests

############ Copy paste of the mail thread:

Hi Lada!

Thanks for the detailed answers, it looks very promising to me. I will try to do some practice exercise with it to get better understanding of how it works.
On 10/5/18 12:43 PM, Ladislav Slezak wrote:

Hello!

Dne 04. 10. 18 v 9:48 Rodion Iafarov napsal(a):

I'll comment the requirements from the libyui REST API prototype POV so we could
see if that would fit and we could compare it with the other possibilities:

These is combined list for the needs for your previous email:

1. Operating on all existing widgets, including tables, trees, comboboxes, tabs. The prototype does not support the complete set of widgets yet (ENOTIME) but adding support for the missing ones is quite easy. The most complex will be the package manager widget as from the YaST POV the whole dialog is a single widget like a single push button. And they are implemented in separate packages (libyui-*-pkg).

Being able to perform same set of actions like user can do normally.
It turned out that the prototype was actually more powerful than the user at the
beginning, e.g. I could change the state of a disabled check box, but user cannot
click it in reality. Later I added a check and disallowed changing the disabled widgets.

But I think this should not be a problem, we just need to think about it, carefully
compare the real UI and REST API possibilities and make it as close as possible.

Josef already mentioned that it's important to check that all widgets are properly
displayed etc... The REST API currently does not provide any information about the
rendered widgets on the screen.

But it should be possible to get some data from the layout engine or the UI itself.
Then we could return additional flags like "out-of-screen" or "partially-displayed"
for each widget and test in the frontend that no widget has these flags set.

1. Possibility to assess widgets properties e.g. values in the table cells, labels
   text, UI properties (if control is visible/disabled)
   The REST API works directly on the libyui level, we can read (almost) everything.
   We can get much more details than it is possible from the YaST code.

2. Some way of integration with openQA (that's important for QA team, as we have
   to use it, but it should not be an issue as we can simply create file with
   results and parse it)
   The integration should be simple, you just need to add some REST API client
   which can be implemented in any language and process the JSON response.

The problem would be when testing a network-less scenario (e.g. network not connected
or even no network card present) because an HTTP REST API naturally requires a network.

But in that case you could connect to the REST API locally via serial console,
run "curl http://localhost/..." there and pass the output further. That looks a bit
annoying but should be solvable quite easy with some wrapper.

1. There is not to complex way to launch tests when running installation (e.g.
   booting from installer DVD and triggering test)
   The prototype is already able to test the installation, you just need to set some
   environment variables via the boot command line to enable the REST API server
   (disabled by default) and that's it. Then the installation can be driven from outside
   via the REST API [1].

2. There is a way to run tests with local changes without configuring any kind of
   complex setup
   It mainly depends on how you design the frontend part of the testing framework, the
   backend part is trivial and once we integrate it into YaST it will run out of box (or
   you just install few packages before using it).

For example the Cucumber prototype is very easy to use, you only need Ruby and
Bundler installed. Install the gems [2], prepare the environment (start a VM if
needed) and you can run the tests [3].

1. Solution can be used to run CI per branch I'm not sure what does "branch" mean in this case. Product branch like GA, SP1, SP2? Could you clarify?

As per one of our discussions we had during YaST workshop, we agreed that we want to test earlier. Ideally, we could apply such CI to run against feature branches, so developers can test their changes even before creating pull request.

LessSome of

1. Can be used for ncurses and qt GUIs
   The prototype already supports both.

2. Running tests minimizes influence on SUT (we should keep difference of real
   installation vs integration one as little as possible not installing hundreds of
   the dependencies).
   So far it just needs two additional packages (libjsoncpp and libmicrohttpd),
   in total it's ~170KB (packed RPMs) so the footprint is really minimal. That's even
   more important for the installer as the inst-sys space is very limited.

In installed system you might additionally need to open the HTTP port at firewall (or
you can use the same trick as with the network-less installation).

Summary of the most important features:

• The REST API can be used from any programming language, it's language independent
• It makes a clean boundary between the frontend and the backend
• The internal JSON format is human readable, we could store the responses for later debugging if needed
• The frontend may use any testing framework (Cucumber, RSpec, ... or a custom one), it can be easily updated/changed
• Small dependencies for the backend
• Works with both Qt and ncurses

[1] https://github.com/lslezak/cucumber-yast#testing-the-yast-installer
[2] https://github.com/lslezak/cucumber-yast#installing-cucumber-ruby-gem
[3] https://github.com/lslezak/cucumber-yast#running-the-tests