action #42503
closedcoordination #40475: [functional][y][saga] Establish YaST team split
coordination #42191: [functional][y][epic] Have separate job group for YaST subteam
[functional][y] Define test matrix for the YaST job group
0%
Description
Motivation¶
When we'll have separate job group, we need good understanding of what we have in it and what we want to have. By that we also can decide where how we want to test one thing and another.
There are many examples where we could be more efficient and gain time for increasing test coverage for more critical areas instead of covering e.g. firefox on all possible environments.
This also includes proper evaluation of the risks when not running certain test module.
As a starting point we already have test suites descriptions + test modules descriptions. But it's hard to identify which modules are executed where, and on which env do we cover certain setups. That is visible only in the openQA dashboard.
Suggestions¶
There is one team which uses testopia exactly for this purpose. We need to identify good format which could be reused among teams and used e.g. as a reporting template to visualize coverage per build.
Markdown is more preferable
Acceptance criteria¶
- All existing test suites in the YaST job group have description
- Test suite includes env description, executed test cases summary and purpose
NOTE: should be handled once we have current state of the job group, meaning after at least #42494 and #42503 are done.
Updated by okurz about 6 years ago
- Category set to Enhancement to existing tests
- Target version set to Milestone 22
Please try to always assign a category
Updated by riafarov about 6 years ago
- Status changed from New to Workable
@okurz: Will do, thanks for the feedback.
Updated by riafarov about 6 years ago
- Status changed from Workable to Feedback
After spending 1 full day I was not able to describe all test suites: https://gitlab.suse.de/riafarov/qa-sle-functional-y/blob/master/SLES_Integration_Level_Testplan.md
As of now, we agreed to gather some feedback before proceeding further. Reviewing current coverage has already helped to identify multiple action points.
Updated by riafarov about 6 years ago
- Due date changed from 2018-12-04 to 2018-12-18
Updated by okurz about 6 years ago
my feedback from the sprint review meeting: A link to the existing test plan would be great (two-way)
also the document mentions "SLES" but we do not cover only SLES, I would extend that to SLE and potentially also include openSUSE
Updated by riafarov about 6 years ago
@okurz, yes, there are quite some parts missing. I have limited scope to YaST job group only for now. Sebastian supports me on this too, as we are trying to solve it not for a single team. Thanks for the feedback!
Updated by riafarov about 6 years ago
- Due date changed from 2018-12-18 to 2019-01-15
Waiting for the feedback from Sebastian.
Updated by riafarov almost 6 years ago
- Due date deleted (
2019-01-15) - Priority changed from Normal to Low
- Target version changed from Milestone 22 to Milestone 23
Keep collecting the feedback to decide how to proceed here.
Updated by riafarov almost 6 years ago
- Status changed from Feedback to Resolved
All scenarios currently enabled in YaST job group are defined: https://gitlab.suse.de/riafarov/qa-sle-functional-y/blob/master/SLES_Integration_Level_Testplan.md
Let's keep it up to date now.