Project

General

Profile

action #71665

coordination #71662: [epic][y] Improve managing/grouping test suites

[y][timeboxed:12h] Research on grouping of test suites in openQA

Added by riafarov 10 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2020-09-22
Due date:
% Done:

0%

Estimated time:

Description

Migration team has used separate flavors to group scenarios. This is a hack, so we should come up with something better.

We need to evaluate what we can do for the grouping. Talk to tools team about possibilities, and potentially to other teams to see how they solve it #71662.

History

#1 Updated by riafarov 10 months ago

  • Subject changed from [y] Research on grouping of test suites in openQA to [y][timeboxed:12h] Research on grouping of test suites in openQA
  • Description updated (diff)
  • Status changed from New to Workable

#2 Updated by JERiveraMoya 10 months ago

  • Assignee set to JERiveraMoya

#3 Updated by riafarov 10 months ago

  • Project changed from openQA Tests to qe-yast
  • Category deleted (Spike/Research)

#4 Updated by JERiveraMoya 10 months ago

  • Status changed from Workable to In Progress

#5 Updated by JERiveraMoya 10 months ago

  • Status changed from In Progress to Resolved

Discussing in our team we came up with two options:
(1) We see that from the UI 'SLE 15' is a folder and 'YaST' is a group but we are guessing that internally both are groups because one is called parent group and the other group, so one way could be to implement more nested items that could be reached out from the top-left menu 'Job Groups' so could select "Job Groups -> SLE 15 -> YaST -> storage" and just view our storage test section. At the same time if we would select "Job Groups -> SLE 15 -> YaST" we would see all together. Here we are guessing that the difficulty would be on tweaking this menu and the rest is already done regarding visualization, in other words to allow more than 2 level of nesting.

(2) Not change previous menu mentioned above and leave the group as it is, so in group page, for example ours we would create some sections inside each flavor (possibly that user can expand or compact, but this is just optional fancy idea). We would have some new tag to use in the test suite to be able to assign test suite to each section and for teams not using this feature the screen would look the same, filter also would work in the way that if section is empty the frame is not visualized. Also we should consider here to user frame inside frames, sections inside sections, so it could scale if in the future for example we have a lot of 'storage' tests, then we can create sections 'storage-ng-v4' and 'storage-ng-v3' inside 'storage' section.


From Tools teams this is the feedback I could gather:
(1) When it was implemented the parent groups it was decided against allowing deeper nesting for simplicity. It would be a big change, indeed, e.g. all places which are currently dealing with job groups and their parents needed to be adjusted (the group editor pages, the group overview, the menu, the cleanup code, …). Especially now that the decision has already been made to support only 2 levels of nesting it would be even hard to revert that limitation. I'd also like to note that the links in the menu currently show the group over view page and not the test overview page. (we were pointed that we were currently talking about the test overview page.)

(2) Adding a category/group/tag name to a test suite would be possible. About the nesting: Test suites can not be nested and it wouldn't be the best idea to introduce nesting there just for displaying purposes. However, we could allow using slashes within the category name so the nesting would work similar to file system paths.

(3) Note that openQA is a big code base and nobody knows everything in detail. Assuming existing hierarchy of the test results overview page is simply deduced from the job settings. So a job is simply shown within a certain "Flavor: …" section if it has a job setting FLAVOR=…. It doesn't really matter where the flavor comes from (test suites, job templates, manually posted job, …). Maybe it makes sense to follow the existing approach, e.g. jobs which have the variable CATEGORY=… would be shown within a sub section "…" of the "Flavor: …" section.


Conclusion:
(1) would requires too many change so basically we can discard.
(2) and (3) which would looks something like this should be a good option:

FLAVOR: Full
   > storage/raid
----------------------------------------
Test           aarch64 ..........
---------------------------------------
|  Test 1
|  Test 2
----------------------------------------
   > storage/lvm
|  Test 3
|  Test 4
---------------------------------------
FLAVOR: Online
...

#6 Updated by riafarov 10 months ago

  • Due date deleted (2020-10-06)

Also available in: Atom PDF