Project

General

Profile

Actions

coordination #36712

closed

[saga] Use YaST specific framework for GUI testing

Added by riafarov over 6 years ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2018-06-13
Due date:
2020-10-20
% Done:

100%

Estimated time:
(Total: 178.00 h)

Description

Motivation

Outcome of https://hackweek.suse.com/16/projects/make-yast-testing-independent-of-keyboard-shortcuts

YaST already has mechanism which may help us to develop more scalable tests using widget IDs instead of needles.

Some parts got broken meanwhile and even simple installation doesn't work when just clicking next.
Idea is to automate one test suite which we already have in openQA and evaluate if it is faster/scalable/easier to maintain/etc.
One of candidates can be yast2_gui/yast2_hostnames.pm . It will require improvements in widgets handling that we can select rows in the table.

Acceptance criteria

  • AC1: Simple YaST module test is automated using GUI framework which is not relying on needles and shortcuts

Conclusion copied:

Conclusion and future steps

Defined goal was reached, pop-up approached was implemented and integrated with openqa to prove the concept. Here is link to the branch with changes: os-autoinst-distri-opensuse During the project I've learned more about yast gui internals. This functionality can be used in the tests with constantly changing keyboard hotkeys in first place.

After seeing this possibility to have more stable tests using yast gui internals, we can use advantage of it and stop relying on needles and keyboard shortcuts. After that I was introduced to YCP functionality which is there for a long time and already has a recorder and a player. Documentation for the Yast Programming language can be found here. This functionality can be used for testing and already contains the functionality to read widget properties and set them, as well as interact with them simulating user input.

YCP macro can be executed when application is started by simply calling: /usr/lib/YaST2/bin/y2base ./HelloWorld.rb qt --macro macro.ycp Or, it can be also called during the run using ctrl-alt-shift-p key combination and then selecting ycp file.

This part makes it possible to operate on lower level and hence make automated tests more stable and scalable.
Pros

  • Usage of widget IDs allows to write tests which are resistant to fonts and layout changes
  • Usage of YCP allows to avoid synchronization issues (when some control is not yet shown, but we try to act on it)
  • YCP allows to avoid send keys problems we experience in openQA
  • Reduced execution time as we are able to set controls property to certain value instead of typing
  • Macro recorder allows to partly reduce test code development
  • Tests developed in this way can be easily executed by developers and significantly reduce feedback cycle
  • Can be combined with needle matching approach
  • Easy migration to the solution (e.g. only newly or unstable developed tests are developed using YCP)

Cons

  • Learning curve for test developers
  • Missing functionality and requires further development (e.g. not possible to verify values in the table or select row with certain cell)
  • Unstable behavior if used with installer (unfortunately due to some bugs installer may crash during macro execution, some trivial actions like pressing button may not work if change installers page, etc.)
  • Missing openQA integration (even though it's easy to run such a test, and log errors in y2log file, the results won't be parsed and displayed in friendly way on the dashboard)
  • Works only with Yast GUI, so usage scope is limited

All the disadvantages are mainly caused by the fact that YCP is not widely used, but that can be solved with reasonable time investment. In order to prove benefits of the solution we could implement one of unstable tests using YCP and run them in parallel to calculate time spent on maintenance of both solutions.


Subtasks 44 (0 open44 closed)

action #37324: [sle][functional][y][hard] Fix YCP framework to conduct simple installationResolvedcwh2018-06-132018-08-14

Actions
coordination #38876: [functional][y][epic] Automate yast2 hostname test suite using YaST framework which is not relying on shortcutsRejectedriafarov2018-06-132019-01-11

Actions
action #37327: [sle][functional][y] Improve YaST GUI testing framework to operate on tablesResolvedcwh2018-06-132018-11-20

Actions
action #38879: [functional][y] Implement yast2 hostname test using ncurses with YCPRejected2018-12-112019-01-11

Actions
action #42188: [functional][y][timeboxed:8h] Evaluate testing YaST GUI framework developed by Ladislav LezakResolvedcwh2018-10-092018-10-23

Actions
coordination #43742: [functional][y][epic] Separate YaST UI framework to individual projectResolvedriafarov2019-01-082019-04-09

Actions
action #45824: [functional][y] split libyuiResolvedriafarov2019-01-08

Actions
action #45827: [functional][y] Split libyui-ncursesResolvedriafarov2019-01-082019-04-09

Actions
action #45830: [functional][y] Split libyui-qtResolvedriafarov2019-01-082019-03-26

Actions
action #49430: [functional][y] Adjust testframework proposals to the feedback and establish packageResolvedriafarov2019-03-18

Actions
coordination #50672: [funtional][y][epic] Use libyui-rest-api for YaST modules testingResolvedJERiveraMoya2019-04-252020-05-05

Actions
action #50726: [functional][y][timeboxed:16h] Explore different ideas which technologies to use with libyui-rest-apiResolvedJERiveraMoya2019-04-252019-07-16

Actions
action #54305: [functional][y][timeboxed:16h] Explore different ideas which technologies to use with libyui-rest-api ResolvedJERiveraMoya2019-04-252019-07-30

Actions
action #54836: [functional][y][timeboxed:4h] Gather feedback on our proposal for integration testing with libyui-restapiResolvedriafarov2019-07-30

Actions
action #56009: [functional][y] Develop ruby gem library to operate controls in the UIResolvedJERiveraMoya2019-08-272019-09-10

Actions
action #56786: [functional][y] Improve ruby gem library to operate controls in the UIResolvedJERiveraMoya2019-09-112019-09-24

Actions
action #57263: [functional][y] Add aruba + make test not destructive to ruby gem library to operate UI controlsResolvedJERiveraMoya2019-09-242019-10-22

Actions
action #58262: [functional][y] Add support for missing controls to libyui-restapiResolvedriafarov2019-10-162019-11-05

Actions
action #59070: [functional][y] Get https://github.com/libyui/libyui-rest-api/pull/3 and related PRs mergedResolvedriafarov2019-11-05

Actions
action #62525: [functional][y] Propose development environment for the rspec test development with libyuiResolvedybonatakis2020-01-222020-02-11

Actions
action #62531: [functional][y][timeboxed:24h] Introduce api version compatibility for the rest-apiResolvedoorlov2020-01-222020-02-11

Actions
action #65381: [functional][y] Add support for YDateField and YTimeFieldResolvedriafarov2020-04-07

Actions
action #65384: [functional][y] Use JSON in all responses from libyui-rest-apiResolvedriafarov2020-04-07

Actions
action #65390: [functional][y] Add ability to enter text for editable YComboBox Resolvedriafarov2020-04-072020-05-05

Actions
action #65100: [functional][y] Changing a value in an element does not trigger events to reload other elements of the window.Resolvedriafarov2019-04-252020-05-05

Actions
action #65396: [functional][y] Move YRichText and YMenuButton out of "press" action blockResolvedriafarov2020-04-07

Actions
action #65459: [functional][y] Content-Encoding header is used incorrectlyResolvedriafarov2020-04-082020-05-05

Actions
coordination #62726: [functional][y][epic] Create separate Ruby Gem representing libyui Client APIClosedriafarov2020-04-072020-10-20

Actions
action #65378: [functional][y] Design LibyuiClient with OOP principlesResolvedoorlov2020-04-072020-04-21

Actions
action #65930: [functional][y] Use enum for actions instead of hard-coded valuesResolved2020-04-21

Actions
action #65936: [functional][y] Create widget classes that are missed in libyui_clientResolvedsyrianidou_sofia2020-04-212020-07-14

Actions
action #65939: [functional][y] Add unit tests for public api methods in libyui_clientResolvedybonatakis2020-04-212020-06-16

Actions
action #65960: [functional][y] Implement libyui_client widgets required to make test for registration moduleResolvedJERiveraMoya2020-04-222020-05-19

Actions
action #66415: [functional][y] Implement libyui_client widgets required to make test for expert partitionerResolvedJERiveraMoya2020-05-052020-06-16

Actions
action #66769: [functional][y] Add support to search widgets using regexp as a filterResolvedoorlov2020-05-132020-06-30

Actions
action #67639: [functional][y][timeboxed:12h] Enable reuse of code for libyui_client widgetsResolvedybonatakis2020-06-032020-06-16

Actions
action #68944: [functional][y] Adjust client code to support new changes on server sideResolvedriafarov2020-07-142020-08-25

Actions
action #70504: [y][timeboxed:20h] Establish build and releases of ruby libyui clientResolvedybonatakis2020-08-252020-09-22

Actions
action #71725: [y] Establish build and releases of ruby libyui clientClosedriafarov2020-09-232020-10-20

Actions
action #67393: [functional][y] Selection of rows does not work as expected when set in JSON in different order than are displayedResolvedriafarov2020-05-28

Actions
action #67396: [sle][functional][y] Support YCheckBoxFrame in server sideResolvedriafarov2020-05-28

Actions
action #68983: [y][timeboxed:24h] Explore possibilities for test written using libyuiResolvedJERiveraMoya2020-07-142020-07-28

Actions
action #69184: [y][timeboxed:24h] Design architecture for ruby rspec yast testsResolvedJERiveraMoya2020-07-212020-08-25

Actions
action #70507: [y] Add support for reading values from YDateField and YTimeFieldResolvedJERiveraMoya2020-08-252020-09-08

Actions
Actions #1

Updated by riafarov over 6 years ago

  • Description updated (diff)
Actions #2

Updated by okurz over 6 years ago

  • Description updated (diff)
  • Due date set to 2018-07-03
  • Category set to New test
  • Target version set to Milestone 17

I guess you would be happy to do it soonish so let's do it, doesn't have to be hackweek because you can reserve hackweek for even more fancy, new, crazy ideas :)

Next sprint?

Actions #3

Updated by riafarov over 6 years ago

I guess we should convert it to the epic and schedule first parts to the next sprint. In general, if one person will spend 2 full weeks on that, we will be able to fulfill acceptance criteria. And for sure next sprint(s) is a good idea not to get stuck in test fixing phase again...

Actions #4

Updated by okurz over 6 years ago

yep, feel free to split into more subtasks and schedule this one for next sprint and follow-ups for the sprints after that.

Actions #5

Updated by riafarov over 6 years ago

  • Subject changed from [sle][functional][y][yast][hackweek] Start using YCP for testing in openQA to [sle][functional][y][yast][hackweek][epic] Start using YCP for testing in openQA
Actions #6

Updated by okurz over 6 years ago

  • Target version changed from Milestone 17 to Milestone 17
Actions #7

Updated by okurz over 6 years ago

  • Due date changed from 2018-07-03 to 2018-12-31

due to changes in a related task

Actions #8

Updated by okurz over 6 years ago

  • Target version changed from Milestone 17 to Milestone 22
Actions #9

Updated by riafarov over 6 years ago

  • Subject changed from [sle][functional][y][yast][hackweek][epic] Start using YCP for testing in openQA to [sle][functional][y][yast][hackweek][saga] Start using YCP for testing in openQA
Actions #10

Updated by riafarov over 6 years ago

  • Due date changed from 2018-07-31 to 2018-12-31

due to changes in a related task

Actions #11

Updated by okurz over 6 years ago

I like the idea of a saga however what I think would be even better if we find a new saga instead which does not prescribe a technical solution, e.g. does not mention YCP but rather something about workflows, team compositions, new focus areas, improvement of product development workflows related to the yast team etc. If you think we only need a saga because a subticket is an epic: I don't have a problem with using epic on multiple levels, we already do that in other cases as well. So it could very well be something like: saga -> epic -> epic -> story

Actions #12

Updated by riafarov over 6 years ago

  • Subject changed from [sle][functional][y][yast][hackweek][saga] Start using YCP for testing in openQA to [sle][functional][y][yast][hackweek][saga] Use YaST specific framework for GUI testing
  • Description updated (diff)
Actions #13

Updated by riafarov over 6 years ago

@okurz, I agree that it's not about YCP at all and it's planned to be dropped anyway. I have adjusted some parts to emphasize that we just need another testing framework for testing YaST GUI applications.

Actions #14

Updated by riafarov almost 6 years ago

  • Due date changed from 2019-01-29 to 2019-01-11

due to changes in a related task

Actions #15

Updated by okurz almost 6 years ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version changed from Milestone 22 to Milestone 24

I think currently the subtasks describe in details what is workable for the next time so tracking the saga as "blocked" by the subtasks. Depending on when we would have them done setting the target version accordingly.

Actions #16

Updated by riafarov almost 6 years ago

  • Due date changed from 2019-03-12 to 2019-01-11

due to changes in a related task

Actions #17

Updated by okurz almost 6 years ago

  • Assignee changed from okurz to riafarov

@riafarov to track sub-tasks as stand-in as discussed.

Actions #18

Updated by riafarov over 5 years ago

  • Status changed from Blocked to In Progress
  • Target version changed from Milestone 24 to Milestone 26
Actions #19

Updated by riafarov over 5 years ago

  • Due date changed from 2019-04-09 to 2019-06-04

due to changes in a related task

Actions #20

Updated by riafarov over 5 years ago

  • Target version changed from Milestone 26 to Milestone 27
Actions #21

Updated by riafarov over 5 years ago

  • Due date changed from 2019-07-30 to 2019-08-27

due to changes in a related task

Actions #22

Updated by mgriessmeier over 5 years ago

  • Target version changed from Milestone 27 to Milestone 28
Actions #23

Updated by riafarov about 5 years ago

  • Target version changed from Milestone 28 to future
Actions #24

Updated by oorlov over 4 years ago

  • Due date changed from 2020-04-21 to 2020-02-11

due to changes in a related task

Actions #25

Updated by oorlov over 4 years ago

  • Due date changed from 2020-04-21 to 2020-05-05

due to changes in a related task

Actions #26

Updated by riafarov over 4 years ago

  • Due date changed from 2020-07-28 to 2020-08-11

due to changes in a related task: #69184

Actions #27

Updated by riafarov over 4 years ago

  • Due date changed from 2020-08-11 to 2020-08-25

due to changes in a related task: #69184

Actions #28

Updated by riafarov about 4 years ago

  • Project changed from openQA Tests (public) to qe-yam
  • Category deleted (New test)
  • Status changed from In Progress to Blocked
Actions #29

Updated by riafarov about 4 years ago

  • Project changed from qe-yam to openQA Tests (public)
Actions #30

Updated by szarate about 4 years ago

  • Tracker changed from action to coordination
  • Status changed from Blocked to New
Actions #32

Updated by riafarov about 4 years ago

  • Project changed from openQA Tests (public) to qe-yam
  • Subject changed from [sle][functional][y][yast][hackweek][saga] Use YaST specific framework for GUI testing to [saga] Use YaST specific framework for GUI testing
Actions #33

Updated by riafarov over 3 years ago

  • Assignee changed from riafarov to oorlov
Actions #34

Updated by oorlov almost 3 years ago

  • Assignee changed from oorlov to JERiveraMoya
Actions #35

Updated by JERiveraMoya 11 months ago

  • Status changed from New to Resolved
Actions

Also available in: Atom PDF