QA » openQA Project

I have linked this ticket to already existing #88229 . I have the feeling that we did not see this need as severly before most of SUSE openQA contributors have selected to use YAML schedule files per openQA scenario in comparison to previously using a shared schedule definition where differences would all be noted down explicitly. IMHO the whole problem of differing coverage was made worse with #54839 which of course helped to mitigate short-term pain because teams "felt" as if they would step less on each other's toes.
Would we need a "tool" to compare coverage in case we would simply use the same schedule definitions by default?

okurz wrote:

To be able to proceed we need actual use cases. For example I wonder: What are the actual goals you want to achieve?

From the description:

The tool should be able to answer two main questions:

What is the coverage / What tests do we run for product X? (i.e. What tests are running on SLE 15 SP2?)
What is the difference in coverage for products X and Y? (i.e. What tests are running on openSUSE Tumbleweed but not on SLE 15 SP2 and vice versa? )
both of these questions were already asked, but it's hard to give quick and precise answer at this moment

I have read the description. But why do you need the coverage? What would you do with this information if you have it?

okurz wrote:

I have read the description. But why do you need the coverage? What would you do with this information if you have it?

Mostly for filling the gaps, checking if all is OK during release of new service pack or even whole new SLES. Also there is a possibility that someone will want to use it for whatever statistics they need.

It seems to be very interesting for product owners, but any team in QE could find it useful.

Just to get expectations aligned: SUSE QE Tools has not much experience with the test distribution os-autoinst-distri-opensuse itself and also not expectations regarding something like the expected test coverage data. AFAIK reading data from multiple instances and comparing against each other has never been done so far. What I could think of being possible here is an external script accessing the databases of each instance directly, reading test modules, sorting test modules by DISTRI, FLAVOR, VERSION, ARCH, MACHINE . I assume what we would end up with is a very big document that can be used for reference and searching for individual test modules. This can help to answer a question like "Is module X tested on Y at all". But I think it will not be usable to effectively compare test coverage to find gaps assuming that the actual difference will be very big. This is the reason why I proposed #91656 assuming that it's easier to implement, easier to use and more helpful in the long run.

In the meantime maybe also https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module can be helpful to find out in which scenarios specified modules are used.

Regarding a time expectation when we could come to implementing the current ticket my current estimate is in the range of months. See the complete current SUSE QE Tools team backlog under https://progress.opensuse.org/issues?query_id=230

I'm fully aware how difficult this issue is. And my personal expectations don't matter, really, as this is coming more from product owners and I'm here just a middleman who created the ticket. As I mentioned in chat, most interested people in this kind of functionality were Marita, Heiko and Timo. I suggest you talk to each other and align expectations, without me introducing needles noise to the discussion.

I understand. Yes, that makes sense. I guess it's best to do that then in #72877

These tickets are not on high prio

@szarate I don't understand why you think that we should reject this topic. The original requirement is still not fulfilled, so why reject?

okurz wrote:

@szarate I don't understand why you think that we should reject this topic. The original requirement is still not fulfilled, so why reject?

perhaps I misunderstood your comment, I guess you were referring to Juraj's discussion?