Project

General

Profile

action #42188

coordination #36712: [saga] Use YaST specific framework for GUI testing

[functional][y][timeboxed:8h] Evaluate testing YaST GUI framework developed by Ladislav Lezak

Added by riafarov about 3 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
SUSE QA - Milestone 20
Start date:
2018-10-09
Due date:
2018-10-23
% Done:

0%

Estimated time:

Description

Motivation

As per discussion on yast-devel mailing thread, it was decided that solution developed by Lada during the hackweek is a good starting point for the development.
It's also clear that we should not any efforts in bringing back macro player/recorder with YCP, but YCP bindings seem to stay for a longer period of time.

We need to get familiar with the changes, to decide how to proceed and estimate efforts, so this solution can be used on the production.

Here are useful links:

  1. https://github.com/lslezak/cucumber-yast#installing-cucumber-ruby-gem
  2. https://hackweek.suse.com/17/projects/yast-integration-tests-using-cucumber
  3. https://github.com/libyui/libyui-qt/tree/http_server

Acceptance criteria

  1. Identify current capabilities and how it can be used for YaST modules integration tests

############ Copy paste of the mail thread:

Hi Lada!

Thanks for the detailed answers, it looks very promising to me. I will try to do some practice exercise with it to get better understanding of how it works.
On 10/5/18 12:43 PM, Ladislav Slezak wrote:

Hello!

Dne 04. 10. 18 v 9:48 Rodion Iafarov napsal(a):

I'll comment the requirements from the libyui REST API prototype POV so we could
see if that would fit and we could compare it with the other possibilities:

These is combined list for the needs for your previous email:

  1. Operating on all existing widgets, including tables, trees, comboboxes, tabs. The prototype does not support the complete set of widgets yet (ENOTIME) but adding support for the missing ones is quite easy. The most complex will be the package manager widget as from the YaST POV the whole dialog is a single widget like a single push button. And they are implemented in separate packages (libyui-*-pkg).

Being able to perform same set of actions like user can do normally.
It turned out that the prototype was actually more powerful than the user at the
beginning, e.g. I could change the state of a disabled check box, but user cannot
click it in reality. Later I added a check and disallowed changing the disabled widgets.

But I think this should not be a problem, we just need to think about it, carefully
compare the real UI and REST API possibilities and make it as close as possible.

Josef already mentioned that it's important to check that all widgets are properly
displayed etc... The REST API currently does not provide any information about the
rendered widgets on the screen.

But it should be possible to get some data from the layout engine or the UI itself.
Then we could return additional flags like "out-of-screen" or "partially-displayed"
for each widget and test in the frontend that no widget has these flags set.

  1. Possibility to assess widgets properties e.g. values in the table cells, labels
    text, UI properties (if control is visible/disabled)
    The REST API works directly on the libyui level, we can read (almost) everything.
    We can get much more details than it is possible from the YaST code.

  2. Some way of integration with openQA (that's important for QA team, as we have
    to use it, but it should not be an issue as we can simply create file with
    results and parse it)
    The integration should be simple, you just need to add some REST API client
    which can be implemented in any language and process the JSON response.

The problem would be when testing a network-less scenario (e.g. network not connected
or even no network card present) because an HTTP REST API naturally requires a network.

But in that case you could connect to the REST API locally via serial console,
run "curl http://localhost/..." there and pass the output further. That looks a bit
annoying but should be solvable quite easy with some wrapper.

  1. There is not to complex way to launch tests when running installation (e.g.
    booting from installer DVD and triggering test)
    The prototype is already able to test the installation, you just need to set some
    environment variables via the boot command line to enable the REST API server
    (disabled by default) and that's it. Then the installation can be driven from outside
    via the REST API [1].

  2. There is a way to run tests with local changes without configuring any kind of
    complex setup
    It mainly depends on how you design the frontend part of the testing framework, the
    backend part is trivial and once we integrate it into YaST it will run out of box (or
    you just install few packages before using it).

For example the Cucumber prototype is very easy to use, you only need Ruby and
Bundler installed. Install the gems [2], prepare the environment (start a VM if
needed) and you can run the tests [3].

  1. Solution can be used to run CI per branch I'm not sure what does "branch" mean in this case. Product branch like GA, SP1, SP2? Could you clarify?

As per one of our discussions we had during YaST workshop, we agreed that we want to test earlier. Ideally, we could apply such CI to run against feature branches, so developers can test their changes even before creating pull request.

LessSome of

  1. Can be used for ncurses and qt GUIs
    The prototype already supports both.

  2. Running tests minimizes influence on SUT (we should keep difference of real
    installation vs integration one as little as possible not installing hundreds of
    the dependencies).
    So far it just needs two additional packages (libjsoncpp and libmicrohttpd),
    in total it's ~170KB (packed RPMs) so the footprint is really minimal. That's even
    more important for the installer as the inst-sys space is very limited.

In installed system you might additionally need to open the HTTP port at firewall (or
you can use the same trick as with the network-less installation).

Summary of the most important features:

  • The REST API can be used from any programming language, it's language independent
  • It makes a clean boundary between the frontend and the backend
  • The internal JSON format is human readable, we could store the responses for later debugging if needed
  • The frontend may use any testing framework (Cucumber, RSpec, ... or a custom one), it can be easily updated/changed
  • Small dependencies for the backend
  • Works with both Qt and ncurses

[1] https://github.com/lslezak/cucumber-yast#testing-the-yast-installer
[2] https://github.com/lslezak/cucumber-yast#installing-cucumber-ruby-gem
[3] https://github.com/lslezak/cucumber-yast#running-the-tests

History

#1 Updated by cwh about 3 years ago

  • Status changed from Workable to In Progress
  • Assignee set to cwh

#2 Updated by cwh about 3 years ago

Using Leap 15 in a VirtualBox VM as a testing system.
Setup of the REST-API enabled libyui is trivial when following the instructions from https://github.com/lslezak/cucumber-yast
Additional note: additional to the zypper dup from the README the package libyui-qt9 needs to be installed. Otherwise the Qt-UI does not work.

#3 Updated by cwh about 3 years ago

Evaluating the REST API:

What currently is possible:

  • inspect the current dialog and see all the widget types and IDs, all their contents, also for text entrieas and tables.
  • trigger actions for most of the widgets, like enter data, toggle check boxes and press buttons.

What is not (yet) possible:

  • send key presses
  • select rows from tables

So it currently is not possible to e.g. delete a specific entry from /etc/hosts using the YaST host module.

Using the following sequence adds a new enty to the known hosts of the hosts module:
curl -X POST 'http://d133.suse.de:14155/widgets?id=add&action=press_button'
curl -X POST 'http://d133.suse.de:14155/widgets?id=host&action=enter_text&value=192.168.0.1'
curl -X POST 'http://d133.suse.de:14155/widgets?id=name&action=enter_text&value=router'
curl -X POST 'http://d133.suse.de:14155/widgets?id=ok&action=press_button'

Being able to use the REST API in os-autoinst tests would be really helpful for sending actions by widget IDs and also to check text contents by string matching.
However it will still be needed to do screen checks by needle matching to make sure that anything that we can see via the API is really visible to the user.

#4 Updated by riafarov about 3 years ago

Just a quick one, all the commands you've tried, do they work in exactly same way as for ncurses? Or action names can be different, etc.?

#5 Updated by cwh about 3 years ago

About inspecting the dialog and contents of widgets:

After filling a text entry using
http://d133.suse.de:14155/widgets?id=name&action=enter_text&value=router

the call

http://d133.suse.de:14155/widgets?id=name

returns

[
{
"class" : "YInputField",
"debug_label" : "Hostname",
"hstretch" : true,
"id" : "name",
"label" : "&Hostname",
"password_mode" : false,
"value" : "router"
}
]

So data entry can easily be verified.

#6 Updated by cwh about 3 years ago

riafarov wrote:

Just a quick one, all the commands you've tried, do they work in exactly same way as for ncurses? Or action names can be different, etc.?

Verified that they exactly work the same.

#7 Updated by cwh about 3 years ago

Useful information can also be found here: https://blog.ladslezak.cz/2018/07/13/hackweek-17/

#8 Updated by okurz about 3 years ago

  • Category set to Enhancement to existing tests

#9 Updated by okurz about 3 years ago

  • Target version set to Milestone 20

#10 Updated by cwh about 3 years ago

  • Category deleted (Enhancement to existing tests)
  • Status changed from In Progress to Resolved
  • Target version deleted (Milestone 20)

#11 Updated by cwh about 3 years ago

  • Category set to Enhancement to existing tests
  • Target version set to Milestone 20

Also available in: Atom PDF