action #91509
open
coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA
coordination #88229: [epic] Prevent unintended test coverage decrease
[spike-solution]Easy way to check and compare coverage in multiple openQA instances
Added by hurhaj over 3 years ago.
Updated about 2 months ago.
Estimated time:
(Total: 0.00 h)
Description
Motivation¶
We have (AFAIK) three official instances of openQA:
- openqa.suse.de
- openqa.qam.suse.cz
- openqa.opensuse.org
To get a full picture of our openQA efforts, we need data from all of them, otherwise we will be missing info about some of the products (e.g. QEM has runs in both o.s.d and o.q.s.c).
The tool should be able to answer two main questions:
- What is the coverage / What tests do we run for product X? (i.e. What tests are running on SLE 15 SP2?)
- What is the difference in coverage for products X and Y? (i.e. What tests are running on openSUSE Tumbleweed but not on SLE 15 SP2 and vice versa? )
- both of these questions were already asked, but it's hard to give quick and precise answer at this moment
## Suggestions
- Come up with one or multiple proposals for how to implement this
- Consider a basic web app which can fetch data from multiple openQA instances
- Could this be implemented as an openQA plugin? Make osd the primary place to look at complete coverage?
- Generate reports in GitLab pipelines a la openqa_review
- Ask PMs such as Heiko and Timo what they would find useful
- Description updated (diff)
- Category set to Feature requests
- Target version set to future
- Parent task set to #88229
I have linked this ticket to already existing #88229 . I have the feeling that we did not see this need as severly before most of SUSE openQA contributors have selected to use YAML schedule files per openQA scenario in comparison to previously using a shared schedule definition where differences would all be noted down explicitly. IMHO the whole problem of differing coverage was made worse with #54839 which of course helped to mitigate short-term pain because teams "felt" as if they would step less on each other's toes.
Would we need a "tool" to compare coverage in case we would simply use the same schedule definitions by default?
- Subject changed from [tools] Create tool for checking and comparing coverage in openQAs to Easy way to check and compare coverage in multiple openQA instances
okurz wrote:
To be able to proceed we need actual use cases. For example I wonder: What are the actual goals you want to achieve?
From the description:
The tool should be able to answer two main questions:
What is the coverage / What tests do we run for product X? (i.e. What tests are running on SLE 15 SP2?)
What is the difference in coverage for products X and Y? (i.e. What tests are running on openSUSE Tumbleweed but not on SLE 15 SP2 and vice versa? )
both of these questions were already asked, but it's hard to give quick and precise answer at this moment
I have read the description. But why do you need the coverage? What would you do with this information if you have it?
- Project changed from openQA Project (public) to openQA Tests (public)
- Category changed from Feature requests to Enhancement to existing tests
- Subject changed from Easy way to check and compare coverage in multiple openQA instances to [tools] Easy way to check and compare coverage in multiple openQA instances
okurz wrote:
I have read the description. But why do you need the coverage? What would you do with this information if you have it?
Mostly for filling the gaps, checking if all is OK during release of new service pack or even whole new SLES. Also there is a possibility that someone will want to use it for whatever statistics they need.
It seems to be very interesting for product owners, but any team in QE could find it useful.
Just to get expectations aligned: SUSE QE Tools has not much experience with the test distribution os-autoinst-distri-opensuse itself and also not expectations regarding something like the expected test coverage data. AFAIK reading data from multiple instances and comparing against each other has never been done so far. What I could think of being possible here is an external script accessing the databases of each instance directly, reading test modules, sorting test modules by DISTRI, FLAVOR, VERSION, ARCH, MACHINE . I assume what we would end up with is a very big document that can be used for reference and searching for individual test modules. This can help to answer a question like "Is module X tested on Y at all". But I think it will not be usable to effectively compare test coverage to find gaps assuming that the actual difference will be very big. This is the reason why I proposed #91656 assuming that it's easier to implement, easier to use and more helpful in the long run.
In the meantime maybe also https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module can be helpful to find out in which scenarios specified modules are used.
Regarding a time expectation when we could come to implementing the current ticket my current estimate is in the range of months. See the complete current SUSE QE Tools team backlog under https://progress.opensuse.org/issues?query_id=230
I'm fully aware how difficult this issue is. And my personal expectations don't matter, really, as this is coming more from product owners and I'm here just a middleman who created the ticket. As I mentioned in chat, most interested people in this kind of functionality were Marita, Heiko and Timo. I suggest you talk to each other and align expectations, without me introducing needles noise to the discussion.
I understand. Yes, that makes sense. I guess it's best to do that then in #72877
These tickets are not on high prio
These tickets are not on high pro
- Tags deleted (
bulkupdate)
- Project changed from openQA Tests (public) to openQA Project (public)
- Category changed from Enhancement to existing tests to Feature requests
- Assignee set to okurz
@szarate I don't understand why you think that we should reject this topic. The original requirement is still not fulfilled, so why reject?
okurz wrote:
@szarate I don't understand why you think that we should reject this topic. The original requirement is still not fulfilled, so why reject?
perhaps I misunderstood your comment, I guess you were referring to Juraj's discussion?
- Project changed from openQA Project (public) to openQA Tests (public)
- Subject changed from [tools] Easy way to check and compare coverage in multiple openQA instances to [qe-core] Easy way to check and compare coverage in multiple openQA instances
- Category changed from Feature requests to Enhancement to existing tests
- Assignee deleted (
okurz)
Well, in any case this isn't done. Moving back to "openQA Tests" as the problem seems to be really self-made due to the decision to have the yaml-based schedule separate per product/version/flavor in os-autoinst-distri-opensuse instead of (my preference) to try to keep the test schedule as much aligned as possible and only apply exceptions, e.g. skip "scc_registration" on openSUSE, skip "steam" on SLE but do the same otherwise for SLE and openSUSE. Also see #91509#note-4 where I have explained this.
- Project changed from openQA Tests (public) to openQA Project (public)
- Subject changed from [qe-core] Easy way to check and compare coverage in multiple openQA instances to Easy way to check and compare coverage in multiple openQA instances
- Category deleted (
Enhancement to existing tests)
- Assignee set to okurz
Please reject the ticket if the tools team doesn't see a valid request, don't reassign to core, as it goes beyond our scope.
Well, I still see this as a valid request though still specific to os-autoinst-distri-opensuse. Let's keep in future for now
- Subject changed from Easy way to check and compare coverage in multiple openQA instances to [spike-solution]Easy way to check and compare coverage in multiple openQA instances
- Description updated (diff)
Also available in: Atom
PDF