QA

Raised priority since review becomes more difficult without this feature

I will clarify before moving on asking the team to conduct work here.

vpelcak wrote:

Hello.

openQA bot was commenting in IBS which lead to massive spam of e-mails.
Besides we also saw opportunities in CI dashboard for the openQA review:
https://progress.opensuse.org/issues/80194 #80194

Therefore some 2 weeks ago we had a meeting - openQA reviewers, Ondrej and me, where openQA reviewers agreed to switch from openQA bot to CI dashboard.

In particular for the point "openQA reviewers, Ondrej and me […] agreed to switch from openQA bot to CI dashboard" this seems to be in conflict with what you expect from me as product owner for SUSE QE Tools and also what you and others expect from the team SUSE QE Tools. I see a problem not only with the technical impacts described above but also with how such a decision was made. One more reason why I am wondering about such a decision which steps would need to be conducted for any "QAM dashboard" first before continuing such changes which impact the process.

The loss of comments was discussed there as well.

And what are the results of the discussion? I am particularly interested as you know that some people still would like to have "notifications".

For SUSE QE maintenance tests we already have a mailing list qa-maintenance-reports@suse.de that receives comments from https://gitlab.suse.de/qa-maintenance/openQABot/ on build.suse.de whenever any incident related tests fail. The archive can be seen on
http://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/index.html . https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.08/maillist.html shows that the last email with "Request … commented by sle-qam-openqa" was from 2021-08-07. I would like to know who applied the change and where the change was applied. Can you clarify that point?

There were no objections from the side of vpelcak and no comment from osukup. In the meantime with #96998 resolved at least the bus factor for bot-ng increased. I have asked lask week in https://chat.suse.de/channel/qem-openqa-review?msg=vuBWpiRhgsJRGqwwj and there was no clear positive nor negative feedback. So I assume we can deprioritize.

Open points I like to clarify:

1. What is the position of mgrifalconi (as he created the ticket initially)
2. how are reports of openQA tests at the time of acceptance provided if not by the comments on IBS/smelt
3. how can we send notification emails when there are new test results regarding incidents

Seeing the feedback from multiple persons I see multiple problems with the approach that was taken which I see as one more reason to go back to the old approach. I simply do not see the "QEM dashboard" ready yet for production and IMHO the impact of relying on it has not been fully envisioned.

As there were no objections from vpelcak (see #97121#note-10) I suggest again to go back to the old ways for now. Putting back to "Workable" for the team to pick up.

cdywan wrote:

I wonder if it would be sensible for this ticket to block on #98601

no, because this ticket is about "bring back old behaviour", which does not need any more detailed specs.

• or alternatively Feedback with an updated Due Date and some idea how to come up with clear ACs. To me it doesn't look "Workable" right now.

Right, we did not estimate as such. To make it more clear I now created an urgent subtask #98637 for the specific task to enable IBS comments again, same as we had in before, and let this ticket stay as an epic which can be used to catch more high-level feedback and to answer additional open points.

Current state¶

With #98637 resolved we have a feature that allows to send IBS comments and hence emails whenever release requests are blocked from auto-approval due to failing tests. This is an important goal reached. However we did not yet enable this feature to automatically and recurringly send comments yet due to objections.

Discussion¶

There was a lengthy discussion in https://suse.slack.com/archives/C02CANHLANP/p1637593271430900 (internal). I am trying to note the important points. Whenever someone writes a comment on IBS that sends emails. This is in general something which I want to have as notification feature but when someone is subscribed to high-traffic projects, many packages, etc., there can be very many emails, referenced as "spam" by some, e.g. coolo, hence a problem. According to coolo this can account for roughly 1k emails per day which is of course a lot. According to coolo originally someone wanted to have IBS comments to "track history" or something. This is certainly not the best way and I don't consider this approach good for that purpose but for different purposes. So maintenance representatives, e.g. Stephan Barth, would like to see comments back on release requests so do I because the submitters learn that their release request is blocked by failing openQA tests. Resending more emails to just transport the same message can deteriorate quickly so maybe a good idea is to only ever send one comment (hence also one email) whenever failing tests block auto-approval and then it is up to submitters and QA engineers to resolve the situation. According to coolo qem-dashboard was started (see https://confluence.suse.com/display/~coolo/Maintenance+CI+Dashboard?focusedCommentId=876905006#comment-876905006) to provide a database that provides track history. According to coolo "All code is between qa-tools members". I consider filtering emails is easier than being able to guess what could possibly block updates when reading IBS comments. coolo stated "https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.11/maillist.html is such a joyful read", likely meaning that the mailing list has high-volume and is hard to read, maybe made worse by the automatic commenting we introduced. However I consider the ML already hard to fully read as even without any automatic comments there are many emails, e.g. "Request XYZ requires review" and similar. So I think automatic comments when tests block release requests are not making the situation worse. Either people manage the email workload with proper processes and filtering or they ignore the emails or unsubscribe. We have other mailing lists which work the same. I stated "And all these 24 individuals and subscribers of the ML are welcome to unsubscribe or filter" and coolo answered "you can't unsubscribe from IBS comments!" which I don't know if that is true but in which case I consider filtering the right approach. coolo: "unless you unsubscribe from all of IBS email, which is not what I want - because I want to be notified about meaningful IBS comments. So you're asking 24 individuals to filter email that shouldn't have been sent to begin with? I review at times - like everyone else. Noone of the receivers reviews full time. But you also send it to all of maint-coord, autobuild and SLE release coord - and those are unreachable by IBS comments now because they did what you recommend: filter all IBS emails to /dev/null. Very few of them will even have bothered to filter the bot comments. And the email you send are even sent as coolo@suse.de, which makes me very valid for objection".
coolo: "Everyone is asking for smelt to show these comments - why not create them in smelt?". okurz: "for two reasons: 1. Because the information is also useful in IBS, 2. because my assumption was that it's easier to first rollback (or bring back) the previous feature before developing something new. Point 2 by now is questionable as it took 3 months (!) to achieve that however that does not mean that it would have been faster to do that in smelt. And that still leaves 1."
coolo: "Why it's useful there? Noone missed it there - or can you point to something I missed? What I would find useful instead: if the bot commented the moment it approved*. Because that's an IBS action - the review history is made for that".
okurz: "there are often questions like: "Why is SR#42 not approved by qam-openqa"? And that comment in IBS would directly give that answer. Alerting 24+ people that the bot looked at something - not so useful". okurz "well, if there would be a comment like "bot: I will not approve because tests failed, take a look on dashboard-url", that would be equivalent, yes. I agree that there are many points to improve but there is one more thing that comments plus the according email provide: A timely trigger. The 'look at the dashboard'-suggestion is good to see the current state at all times but IMHO not the best answer to everything. I also receive many other emails by OBS, IBS, github and most of them I use as a notifcation or a trigger point. I don't need to get all the information from email but at least the 'something happened' event.". coolo: "That's the reason the comment functionality was implemented for openSUSE. we had like 3 updates in the queue and got around 10 emails a day - everytime a trigger to go to openqa and check (and retrigger). but with SLE and IBS all that's left for you is: filter the mess into a folder you have on short delete expire. And the comment isn't really state of the art either". okurz: "Yes, everybody can choose that but what would be the answer to people that look for updates in a specific release request when there would be no emails? Poll a webUI manually? The comment could say 'I finished testing and I could not approve. For the current state go to http://...'.". coolo: "Again: I'm fine with having one comment once it decided it's not good enough. Just like I had initially in mind that the bot comments the final state when it does approve (e.g. the snapshot it used to approve it, but also the exceptions applied). I'm ok with having an IBS comment in 2 situations: the first time all tests finished (pointing to the dashboard only) and the moment the review is approved (with the current summary). So later on it can be verified what the bot actually looked at. Because once it approved the request, it's 'too late' and the machinery is rolling. If it turns out, the bot was buggy or didn't test product X we have to know that in the post mortem"
okurz: "But we want comments in the case of "there is at least one failed" test that blocks auto-approval. Would it help if we somehow say 'as soon as there is one comment on a release request we will never write one again'? I also told every QE squad that they can look at the mailing list same as the dashboard to look for tests that concern them"

Next steps¶

So I suggest the next step that we try is to only write a single comment per release request whenever either a first blocking failure was encountered or all currently scheduled tests for one release request have finished testing and there is at least one blocking failure. Also in the comment point to the current test results, e.g. something like an https://openqa.suse.de/tests/overview link

It seems I have new information. Apparently one can customize if or what notifications are sent. On the page https://build.suse.de/my/subscriptions I found

which looks like the exact thing that coolo stated wasn't possible. Well, maybe it is just possible now. So with this I suggest we simply enable the old comments as was implemented and everybody can unsubscribe for the specific group "qam-openqa" based on their individual preferences. Any objections? If not I will ask for the comments to be enabled again after a waiting time of 14 days, that is 2022-03-10.

Discussed in weekly SUSE QE Tools meeting 2022-04-08. I will announce in some Slack channels that we will enable comments again.

@jbaier_cz as discussed please add the according gitlab CI job to enable comments again.

Announced in

• #discuss-maintenance
• #team-lsg-qe-openqa-review
• #eng-testing
• openqa@suse.de

@jbaier_cz I moved the "enable again" specific task to #109701

coolo wrote:

And compare to https://dashboard.qam.suse.de/incident/23521 giving you a live view on the past days and why they failed for the reviewer to judge if these failures were related to mozilla-nss... which is what #23 asked for.

Yes, you stated already that in comment 23 - reference as #97121#note-23, not #23 – the dashboard would have implemented something. Can you please explain that in more detail? I don't know what you mean.

jbaier_cz wrote:

okurz wrote:

Following the first link I can find 8 scheduled jobs on https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1 not 22 failing tests.

However there were exactly 22 "is waiting" tests. I would guess, this is because the state is not recognized in https://github.com/openSUSE/qem-bot/blob/master/openqabot/commenter.py#L51 so the bot assumes it is a failure.

Maybe. But it shows that the situation is worse than I expected. Also the main problem I see is that there should have only been a comment at all if the auto-approval is blocked which at that point was not the case because there was not a single real failure that would block auto-approval. We would have needed to wait for jobs to complete before making such call.

need to define subtasks.

Waiting first for #107923 which might have a very slim chance of being related. Also I am actually moving the epic out of our backlog again for now. We have more pressing issues and other sagas to finish. The parent epic for this ticket #99303 isn't within our current backlog anyway.