action #88536
closedopenQA Project (public) - coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA
openQA Project (public) - coordination #88229: [epic] Prevent unintended test coverage decrease
Find out differences in openQA test coverage with metabase
Description
User story¶
As a QE engineer, I would like to have a way of comparing openQA's tests (*.pm) utilization between our products, mostly before their release and after the release. With ~1500 tests available at this moment, it's very difficult to find out whether particular test is being used to it's full potential, if it is running on all products, code streams and architectures that it could and should run on. That brings a risk of needlessly lowering the testing coverage that is already available to us.
Acceptance criteria¶
- AC1: Possibility to filter the data, to easily compare coverage between whichever products, code streams.
Tasks¶
- Get accustomed to https://maintenance-statistics.dyn.cloud.suse.de/
- Identify which data we should include besides already mentioned tests, products, code streams and architecture.
- Create a table that can visualize the above
Further details¶
As per previous discussions, Metabase seems to be the best way to realize this.
Files
| which_openQA_testsuites_and_test_modules_fail_the_most_on_OSD_metabase.png (83.2 KB) which_openQA_testsuites_and_test_modules_fail_the_most_on_OSD_metabase.png |
Updated by okurz almost 4 years ago
- Related to action #88127: [tools][qem] Test coverage DB for maintenance updates added
Updated by okurz almost 4 years ago
- Related to coordination #88229: [epic] Prevent unintended test coverage decrease added
Updated by hurhaj almost 4 years ago
Updated by okurz almost 4 years ago
- Subject changed from Comparable test coverage to Find out differences in openQA test coverage with metabase
- Description updated (diff)
- Status changed from Feedback to Workable
- Assignee changed from okurz to hurhaj
- Target version changed from Ready to future
- Parent task set to #88229
@hurhaj I have moved some content into the epic #88229 and set that ticket as "parent" to this one. So then I adapted this ticket to be specific about metabase. By this I think it should be easier to followup with this one specific aspect and leave everything more generic for the parent epic. So I would assume that some parts or all you can solve yourself by getting familiar with metabase. And if you find data missing we could look into providing such data to metabase. Ok?
Updated by hurhaj almost 4 years ago
okurz wrote:
@hurhaj I have moved some content into the epic #88229 and set that ticket as "parent" to this one. So then I adapted this ticket to be specific about metabase. By this I think it should be easier to followup with this one specific aspect and leave everything more generic for the parent epic. So I would assume that some parts or all you can solve yourself by getting familiar with metabase. And if you find data missing we could look into providing such data to metabase. Ok?
yes, if the data will be availabe in metabase, filtering/comparing/etc can be easily done by anyone. thanks for merging
Updated by hurhaj almost 4 years ago
The two available tables in metabase provide some info, but not quite what we need. job_modules contain "script" and jobs contains "distri" and "version". What is needed is table that will look something like:
| grub_test.pm | sle | 15-sp1, 15, ...|
Updated by hurhaj almost 4 years ago
- Status changed from Workable to Resolved
OK, I created:
https://maintenance-statistics.dyn.cloud.suse.de/question/287
which meets my needs for comparing coverage of selected codestreams.
Updated by okurz almost 4 years ago
It is great to hear that you have found a solution. I would be very interested to see how it looks but unfortunately I am missing permissions to access the above URL. Can you change permissions or something? Also I would be interested to hear more about it and also try to cover the question how we can extend on from that and maybe help the other tickets we have?
Updated by hurhaj almost 4 years ago
Since I did not find any permissions, I moved it to https://maintenance-statistics.dyn.cloud.suse.de/collection/82
It's just a joint table of already existing job_modules and jobs, with columns that I need selected. For this poo it's enough.
As for other poos, I welcome anyone to add whichever columns they need and save it. But the problem is that it tends to time out pretty often, especially when I try to filter it, so we will very likely have to contact admins about it. Another thing is that resulting json file is freaking massive, about million lines, so filtering is needed (coming back to time out issue) for any sensible working with the available data.
Updated by maritawerner almost 4 years ago
AS QE Project Manager I would not only be interested in all products, code streams and architectures of a testcase but I would also be interested in the functional area of the specific product, e.g. I would be interested in the area of
- yast
- virtualization
- security
- Kernel
- Dektop
- Network
Pretty similar to what we have in the openQa dashboard for product QA.
Updated by okurz almost 4 years ago
Well, with the job group name all the relevant data should be available in metabase already as well. "metabase" is meant to be easily consumable so you might be able to answer your questions from metabase itself similar to what hurhaj has explained above.
Updated by okurz over 3 years ago
- File which_openQA_testsuites_and_test_modules_fail_the_most_on_OSD_metabase.png which_openQA_testsuites_and_test_modules_fail_the_most_on_OSD_metabase.png added
A more recent example of what can be done with metabase:
For me the queries execute rather quickly, just because the issue was mentioned once that metabase on openQA data would be slow or something.