coordination #36712
closed[saga] Use YaST specific framework for GUI testing
100%
Description
Motivation¶
Outcome of https://hackweek.suse.com/16/projects/make-yast-testing-independent-of-keyboard-shortcuts
YaST already has mechanism which may help us to develop more scalable tests using widget IDs instead of needles.
Some parts got broken meanwhile and even simple installation doesn't work when just clicking next.
Idea is to automate one test suite which we already have in openQA and evaluate if it is faster/scalable/easier to maintain/etc.
One of candidates can be yast2_gui/yast2_hostnames.pm . It will require improvements in widgets handling that we can select rows in the table.
Acceptance criteria¶
- AC1: Simple YaST module test is automated using GUI framework which is not relying on needles and shortcuts
Conclusion copied:¶
Conclusion and future steps
Defined goal was reached, pop-up approached was implemented and integrated with openqa to prove the concept. Here is link to the branch with changes: os-autoinst-distri-opensuse During the project I've learned more about yast gui internals. This functionality can be used in the tests with constantly changing keyboard hotkeys in first place.
After seeing this possibility to have more stable tests using yast gui internals, we can use advantage of it and stop relying on needles and keyboard shortcuts. After that I was introduced to YCP functionality which is there for a long time and already has a recorder and a player. Documentation for the Yast Programming language can be found here. This functionality can be used for testing and already contains the functionality to read widget properties and set them, as well as interact with them simulating user input.
YCP macro can be executed when application is started by simply calling: /usr/lib/YaST2/bin/y2base ./HelloWorld.rb qt --macro macro.ycp Or, it can be also called during the run using ctrl-alt-shift-p key combination and then selecting ycp file.
This part makes it possible to operate on lower level and hence make automated tests more stable and scalable.
Pros
- Usage of widget IDs allows to write tests which are resistant to fonts and layout changes
- Usage of YCP allows to avoid synchronization issues (when some control is not yet shown, but we try to act on it)
- YCP allows to avoid send keys problems we experience in openQA
- Reduced execution time as we are able to set controls property to certain value instead of typing
- Macro recorder allows to partly reduce test code development
- Tests developed in this way can be easily executed by developers and significantly reduce feedback cycle
- Can be combined with needle matching approach
- Easy migration to the solution (e.g. only newly or unstable developed tests are developed using YCP)
Cons
- Learning curve for test developers
- Missing functionality and requires further development (e.g. not possible to verify values in the table or select row with certain cell)
- Unstable behavior if used with installer (unfortunately due to some bugs installer may crash during macro execution, some trivial actions like pressing button may not work if change installers page, etc.)
- Missing openQA integration (even though it's easy to run such a test, and log errors in y2log file, the results won't be parsed and displayed in friendly way on the dashboard)
- Works only with Yast GUI, so usage scope is limited
All the disadvantages are mainly caused by the fact that YCP is not widely used, but that can be solved with reasonable time investment. In order to prove benefits of the solution we could implement one of unstable tests using YCP and run them in parallel to calculate time spent on maintenance of both solutions.
Updated by okurz over 6 years ago
- Description updated (diff)
- Due date set to 2018-07-03
- Category set to New test
- Target version set to Milestone 17
Updated by riafarov over 6 years ago
- Subject changed from [sle][functional][y][yast][hackweek] Start using YCP for testing in openQA to [sle][functional][y][yast][hackweek][epic] Start using YCP for testing in openQA
Updated by riafarov over 6 years ago
- Subject changed from [sle][functional][y][yast][hackweek][epic] Start using YCP for testing in openQA to [sle][functional][y][yast][hackweek][saga] Start using YCP for testing in openQA
Updated by riafarov over 6 years ago
- Subject changed from [sle][functional][y][yast][hackweek][saga] Start using YCP for testing in openQA to [sle][functional][y][yast][hackweek][saga] Use YaST specific framework for GUI testing
- Description updated (diff)
Updated by okurz almost 6 years ago
- Status changed from New to Blocked
- Assignee set to okurz
- Target version changed from Milestone 22 to Milestone 24
Updated by riafarov over 5 years ago
- Status changed from Blocked to In Progress
- Target version changed from Milestone 24 to Milestone 26
Updated by mgriessmeier over 5 years ago
- Target version changed from Milestone 27 to Milestone 28
Updated by riafarov over 4 years ago
- Project changed from openQA Tests (public) to qe-yam
- Category deleted (
New test) - Status changed from In Progress to Blocked
Updated by szarate about 4 years ago
- Tracker changed from action to coordination
- Status changed from Blocked to New
Updated by riafarov about 4 years ago
- Project changed from openQA Tests (public) to qe-yam
- Subject changed from [sle][functional][y][yast][hackweek][saga] Use YaST specific framework for GUI testing to [saga] Use YaST specific framework for GUI testing