QA » openQA Project

I would like to give it a more positive vibe. The goal is to motivate test maintainers to reach a lower false-positive rate and for that I would start with a higher rate of automatically accepted SLE maintenance release requests. As openQA job results are soon pruned, especially passed tests reading such data from openQA might give a too negative impression. Would it be possible to read the number of incidents from smelt where the qam-openqa review was approved automatically and not a human reviewer?

This seems a lot like #96191. Can you disambiguate where this is different? Should this perhaps be an epic?

cdywan wrote:

This seems a lot like #96191. Can you disambiguate where this is different? Should this perhaps be an epic?

#96191 is more concerned about mm tests, as a response to a specific problem (mm tests failing more frequently) while this one concerns overall failure rate and additional data for more general use.
I think turning this to an epic is a good idea. @okurz what's your take?

VANASTASIADIS wrote:

cdywan wrote:

This seems a lot like #96191. Can you disambiguate where this is different? Should this perhaps be an epic?

#96191 is more concerned about mm tests, as a response to a specific problem (mm tests failing more frequently) while this one concerns overall failure rate and additional data for more general use.
I think turning this to an epic is a good idea. @okurz what's your take?

My take on it is #98604#note-1 . Not sure if cdywan has read it or we wrote comments at roughly the same time. So the differences to #96191 I see are:

• #96191 is about failure rate of multi-machine tests, #98604 is about rate of automatically approved SUSE SLE QA maintenance incident release requests
• The goal of #96191 is to monitor the infrastructure with the tools team as target audience, goal of #98604 is to motivate test maintainers to reduce rate of false positives
• #96191 is likely best solved with SQL queries ran by telegraf on OSD, #98604 is likely best solved by reading data from smelt

okurz wrote:

VANASTASIADIS wrote:

cdywan wrote:

This seems a lot like #96191. Can you disambiguate where this is different? Should this perhaps be an epic?

#96191 is more concerned about mm tests, as a response to a specific problem (mm tests failing more frequently) while this one concerns overall failure rate and additional data for more general use.
I think turning this to an epic is a good idea. @okurz what's your take?

My take on it is #98604#note-1 . Not sure if cdywan has read it or we wrote comments at roughly the same time. So the differences to #96191 I see are:

• #96191 is about failure rate of multi-machine tests, #98604 is about rate of automatically approved SUSE SLE QA maintenance incident release requests
• The goal of #96191 is to monitor the infrastructure with the tools team as target audience, goal of #98604 is to motivate test maintainers to reduce rate of false positives
• #96191 is likely best solved with SQL queries ran by telegraf on OSD, #98604 is likely best solved by reading data from smelt

Thank you, Oli. I think it would help a lot if we can not refer to both as "tests". Meaning give this ticket a better title and also be clear in the description, which currently talks about "tests" in a way that can apply to both tickets.

Update and results so far:

Before posting the results, after talking with some of the reviewers, one important note: for the past couple of months, the workflow of openqa-reviewers has changed, resulting in drastic changes in the results as well. Namely:

• Before, they would review failing tests and them approve them manually
• Now, they ping the test developers when failures appear, so that they fix the tests, and wait for the bot to automatically approve them in the next run. They only approve manually in special cases

That drastically reduces the amount of manually approved requests. It doesn't necessarily mean that the failures are reduced, but since the reviewers notify the test developers (or even possibly fix the tests themselves) and then wait for the automatic approval, it's harder to see how many tests are actually failing.

Case in point:

PERIOD 01/01/2021 - 01/08/2021
------------------------------
REQUESTS | APPROVED_BY
------------------------------
1000       real users (manual approval)
 699       sle-qam-openqa (automatic approval)

PERIOD 01/08/2021 - 24/09/2021
------------------------------
REQUESTS | APPROVED_BY
------------------------------
105        others
444        sle-qam-openqa

The new approach is potentially better, as it drives people to fix and improve tests when failures appear. However, looking at the percentage of auto-approvals no more hints to the state of the tests, since they don't happen anymore when failures appear, but are only used as a last resort.

Further data upcoming.

VANASTASIADIS wrote:

Update and results so far:
[…]

thank you for providing such detailed explanation and discussion of the results

Case in point:

PERIOD 01/01/2021 - 01/08/2021
------------------------------
REQUESTS | APPROVED_BY
------------------------------
1000       real users (manual approval)
 699       sle-qam-openqa (automatic approval)

PERIOD 01/08/2021 - 24/09/2021
------------------------------
REQUESTS | APPROVED_BY
------------------------------
105        others
444        sle-qam-openqa

is "others" corresponding to real users then?

So the auto-approval ratio before 2021-08-01 is 0.70 (699/1000) and after that 4.23 . That seems like a way too high improvement :) It's interesting to see that there had been nearly as many sle-qam-openqa approvals in just 2 months compared to 7 months in before?

@okurz thank you, yes, 'others' is referring to real users as in the first example.

About your last paragraph, indeed the change is big. Keep in mind however that the much higher rate of auto approval lately doesn't necessarily mean that tests fail less per se, but that now instead of manually approving requests, the tests are fixed and then the requests get auto-approved in the next run. This probably helps with fixing tests (and makes the reviewing procedure more streamlined). On the other hand it also means auto-approvals won't reflect the test failure rate accurately anymore.

VANASTASIADIS wrote:

[…] the much higher rate of auto approval lately [means] that now instead of manually approving requests, the tests are fixed and then the requests get auto-approved in the next run.

That would be lovely and is what I want to reach as well but I doubt it :)

On the other hand it also means auto-approvals won't reflect the test failure rate accurately anymore.

Same as in before the "manual/automatic approval"-ratio would tell us because in the end if tests are not fixed then someone asks for manual approval. This for example just happened the past days in https://suse.slack.com/archives/C02D16TCP99/p1632753957114700 regarding https://smelt.suse.de/incident/21254/ . Despite me fixing some tests there was at least one blocking test failure. Then a reviewer manually approved the update and after asking back they stated that the decision was made in a verbal meeting with no written down information about that decision (IMHO suboptimal).

ok, for convenience I can link in the images for display directly:

• 2020-2021
  

• 2021
  

The graph 2020-2021 looks like the auto-approval ratio was increasing already since the mid of 2020, interesting.

Thank you for providing the query in a way reusable way.

• What would be necessary to provide the access to the smelt DB to other users, e.g. members of SUSE QE Tools?
• As SMELT seems to be connected somewhat to https://maintenance-statistics.dyn.cloud.suse.de/browse/3 already, how about we try to include queries in there?

EDIT: I tried to include it in https://maintenance-statistics.dyn.cloud.suse.de/question/329?group=qam-openqa&start_date=2021-01-01&end_date=2021-09-30 but I get "ERROR: permission denied for table auth_user", maybe we can find a way to not need that table, maybe just hardcode the id?

• You are correct, indeed the SMELT database can be accessed via metabase.
• You are also correct that some tables (for example the auth_user one) are not accessible there, for privacy/security reasons. Most tables are accessible, though.

For most queries this won't be a problem. For this specific query, the users were needed since the openqa-review bot exists as a user in smelt, and to see what it does you need to be able to at least filter by it.

Maybe there's a way for that to be bypassed, I didn't try much tbh - once I saw I couldn't access the table in metabase I just queried directly. I will look at it.

if you could just lookup the user id of the bot user then I guess we can use that

@okurz here is an implementation using the ids of qam-openqa group and sle-qam-openqa user:
https://maintenance-statistics.dyn.cloud.suse.de/question/330

Thanks Oliver, no problem at all. You can alter and use the queries however you see fit. Thank you for updating the other ticket.