Project

General

Profile

coordination #97121

coordination #99303: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, approval and release

[epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)

Added by mgrifalconi 9 months ago. Updated 15 days ago.

Status:
Workable
Priority:
Normal
Assignee:
Target version:
Start date:
2021-09-14
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Motivation

As a maintenance release coordinator I am being notified if there is any failing openQA test blocking automatic approval of a SLE maintenance update

Acceptance criteria

  • AC1: As soon as testing for an individual release request is finished and if there is at least one failing openQA test a comment is written in IBS informing about the failing openQA tests
  • AC2: If there is no failing openQA test related to an individual release request no comment is written
  • AC3: Only a single comment is ever written on a release request

Suggestions

  • Within qem-bot we already have the feature to send out comments but it seems so far it does not look at the state of openQA jobs so it writes a comment for all release requests whenever triggered which means informing even about all currently running jobs. Maybe the next best task is to actually look at the state and only inform about failing openQA tests
  • Think about moving the trigger point of sending a comment into the approval step or something so when no automatic approval is done instead a comment is written
  • Ensure that only a single comment is written, not multiple whenever qem-bot is called

Further details

Original motivation:
While the qem dashboard is agreed to not yet be productive, smelt comments are very useful while doing manual approval of updates.

I would like to have such resource back, at least until https://progress.opensuse.org/issues/97118 #97118 is sorted out.


Subtasks

action #98637: [timeboxed:20h] try to enable comments on IBS (and smelt) again from SUSE QA maintenance openQA test results size:MResolvedjbaier_cz

action #109701: enable qem-bot comments on IBS again after subscriptions can be personally configuredResolvedjbaier_cz


Related issues

Related to QA - action #96998: Increase bus factor for bot-ng size:MResolved2021-07-08

History

#1 Updated by mgrifalconi 9 months ago

  • Priority changed from Normal to Urgent

Raised priority since review becomes more difficult without this feature

#2 Updated by okurz 9 months ago

  • Project changed from openQA Project to QA
  • Subject changed from enable bot comments on smelt again to enable qa-maintenance/openQABot comments on smelt again
  • Target version set to Ready

#3 Updated by vpelcak 9 months ago

Hello.

openQA bot was commenting in IBS which lead to massive spam of e-mails.
Besides we also saw opportunities in CI dashboard for the openQA review:
https://progress.opensuse.org/issues/80194 #80194

Therefore some 2 weeks ago we had a meeting - openQA reviewers, Ondrej and me, where openQA reviewers agreed to switch from openQA bot to CI dashboard.
The loss of comments was discussed there as well.

Further usability improvements could be discussed with Sebastian Riedel.

#4 Updated by okurz 9 months ago

  • Description updated (diff)

#5 Updated by okurz 9 months ago

  • Related to action #96998: Increase bus factor for bot-ng size:M added

#6 Updated by okurz 9 months ago

  • Assignee set to okurz

I will clarify before moving on asking the team to conduct work here.

#7 Updated by okurz 9 months ago

  • Status changed from New to Feedback

vpelcak wrote:

Hello.

openQA bot was commenting in IBS which lead to massive spam of e-mails.
Besides we also saw opportunities in CI dashboard for the openQA review:
https://progress.opensuse.org/issues/80194 #80194

Therefore some 2 weeks ago we had a meeting - openQA reviewers, Ondrej and me, where openQA reviewers agreed to switch from openQA bot to CI dashboard.

In particular for the point "openQA reviewers, Ondrej and me […] agreed to switch from openQA bot to CI dashboard" this seems to be in conflict with what you expect from me as product owner for SUSE QE Tools and also what you and others expect from the team SUSE QE Tools. I see a problem not only with the technical impacts described above but also with how such a decision was made. One more reason why I am wondering about such a decision which steps would need to be conducted for any "QAM dashboard" first before continuing such changes which impact the process.

The loss of comments was discussed there as well.

And what are the results of the discussion? I am particularly interested as you know that some people still would like to have "notifications".

For SUSE QE maintenance tests we already have a mailing list qa-maintenance-reports@suse.de that receives comments from https://gitlab.suse.de/qa-maintenance/openQABot/ on build.suse.de whenever any incident related tests fail. The archive can be seen on
http://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/index.html . https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.08/maillist.html shows that the last email with "Request … commented by sle-qam-openqa" was from 2021-08-07. I would like to know who applied the change and where the change was applied. Can you clarify that point?

#9 Updated by okurz 9 months ago

So if I understand correctly osukup disabled the schedules for "full" and "incident" tests and instead scheduled them on his workstation manually using https://gitlab.suse.de/qa-maintenance/bot-ng, a personal internal prototype. I would prefer to run "full" tests on qa-maintenance/openQABot again until qa-maintenance/bot-ng is production ready with team responsibility at least after #96998 before we adjust the schedule again. I will ask vpelcak and osukup about the feasibility.

#10 Updated by okurz 9 months ago

  • Due date set to 2021-09-13
  • Priority changed from Urgent to Normal

There were no objections from the side of vpelcak and no comment from osukup. In the meantime with #96998 resolved at least the bus factor for bot-ng increased. I have asked lask week in https://chat.suse.de/channel/qem-openqa-review?msg=vuBWpiRhgsJRGqwwj and there was no clear positive nor negative feedback. So I assume we can deprioritize.

Open points I like to clarify:

  1. What is the position of mgrifalconi (as he created the ticket initially)
  2. how are reports of openQA tests at the time of acceptance provided if not by the comments on IBS/smelt
  3. how can we send notification emails when there are new test results regarding incidents

#11 Updated by VANASTASIADIS 8 months ago

I agree with okurz on this, there were complaints about the loss of comments from other sides as well, as this leads to the gear icons in smelt not pointing to the existing tests.

Can we discuss about restoring that functionality? If this info exists but in another place (i.e. the dashboard) we should at least provide the link on smelt to make testers aware.

#12 Updated by bschmidt 8 months ago

I'd like to chime in here. The link from https://maintenance.suse.de/ to the corresponding test in openQA was quite convenient as long as it where there. So, please restore or replace the now missing functionality.
Just for reference:
https://suse.slack.com/archives/C02D12FF0H1/p1631087431032900

#13 Updated by osukup 8 months ago

maybe the acceptable solution for this feature request will be https://gitlab.suse.de/opensuse/qem-dashboard/-/issues/15 and then use information from qem-dashboard in SMELT

#15 Updated by okurz 8 months ago

  • Status changed from Feedback to Workable
  • Assignee deleted (okurz)
  • Priority changed from Normal to High

Seeing the feedback from multiple persons I see multiple problems with the approach that was taken which I see as one more reason to go back to the old approach. I simply do not see the "QEM dashboard" ready yet for production and IMHO the impact of relying on it has not been fully envisioned.

As there were no objections from vpelcak (see #97121#note-10) I suggest again to go back to the old ways for now. Putting back to "Workable" for the team to pick up.

#17 Updated by okurz 8 months ago

Discussed this with osukup, VANASTASIADIS, jbaier_cz and I see the following three alternatives:

  1. Go back to openQABot which is currently the only component supporting commenting on IBS. According to osukup this would need changes again in code and "metadata" for public cloud as some changes have been done there but not synced back
  2. Go forward with https://gitlab.suse.de/tools/smelt/-/issues/724, i.e. reading out the data that bot-ng puts into qem dashboard. What is certainly not so nice is that the decision to switch to bot-ng and not have IBS comments broke existing workflows and now pressuring smelt developers to use the data from the new source.
  3. Implementing IBS commenting into bot-ng. This would bring back comments (which of course can again annoy some users that don't want them) but still allows smelt to read the data

Regarding the data in qem dashboard I understood that in case the database would be lost likely bot-ng can push all relevant data into an empty database again. The only bad impact in that case would be that potentially unnecessary reschedules of openQA tests would happen which should be acceptable in case of disaster recovery scenarios.

With this I suggest to go for alternative 3., implementing IBS commenting into bot-ng, with a switch to be able to activate/deactivate. What are the opinions of others? Please state your opinions.

#18 Updated by mgrifalconi 8 months ago

Hello,

While I still question the hard switch to a new tool that do not cover all the features of the old one, I agree on moving forward instead of rollback.
I have no preference on where I can see the data SMELT or dashboard, as long as it is there (or even better, the bot would be able to process in my place, see https://progress.opensuse.org/issues/97274 #97274).

I would be curious to hear from other stakeholders as well, like other reviewers.

#19 Updated by okurz 8 months ago

Regarding the proposed solution nr 2. I wonder how notifications can be triggered as soon as tests blocking auto-approval fail as I don't want to rely on humans periodically polling dashboards. Enabling comments sent again would cover that point but maybe I am missing something and there is a better way of pushing the information event-based? Also asked in https://suse.slack.com/archives/C02D16TCP99/p1631607307044800

#20 Updated by cdywan 8 months ago

I wonder if it would be sensible for this ticket to block on #98601 - or alternatively Feedback with an updated Due Date and some idea how to come up with clear ACs. To me it doesn't look "Workable" right now.

#21 Updated by okurz 8 months ago

  • Tracker changed from action to coordination
  • Subject changed from enable qa-maintenance/openQABot comments on smelt again to [epic] enable qa-maintenance/openQABot comments on smelt again
  • Status changed from Workable to Blocked
  • Assignee set to okurz

cdywan wrote:

I wonder if it would be sensible for this ticket to block on #98601

no, because this ticket is about "bring back old behaviour", which does not need any more detailed specs.

  • or alternatively Feedback with an updated Due Date and some idea how to come up with clear ACs. To me it doesn't look "Workable" right now.

Right, we did not estimate as such. To make it more clear I now created an urgent subtask #98637 for the specific task to enable IBS comments again, same as we had in before, and let this ticket stay as an epic which can be used to catch more high-level feedback and to answer additional open points.

#22 Updated by okurz 6 months ago

  • Status changed from Blocked to Feedback

Current state

With #98637 resolved we have a feature that allows to send IBS comments and hence emails whenever release requests are blocked from auto-approval due to failing tests. This is an important goal reached. However we did not yet enable this feature to automatically and recurringly send comments yet due to objections.

Discussion

There was a lengthy discussion in https://suse.slack.com/archives/C02CANHLANP/p1637593271430900 (internal). I am trying to note the important points. Whenever someone writes a comment on IBS that sends emails. This is in general something which I want to have as notification feature but when someone is subscribed to high-traffic projects, many packages, etc., there can be very many emails, referenced as "spam" by some, e.g. coolo, hence a problem. According to coolo this can account for roughly 1k emails per day which is of course a lot. According to coolo originally someone wanted to have IBS comments to "track history" or something. This is certainly not the best way and I don't consider this approach good for that purpose but for different purposes. So maintenance representatives, e.g. Stephan Barth, would like to see comments back on release requests so do I because the submitters learn that their release request is blocked by failing openQA tests. Resending more emails to just transport the same message can deteriorate quickly so maybe a good idea is to only ever send one comment (hence also one email) whenever failing tests block auto-approval and then it is up to submitters and QA engineers to resolve the situation. According to coolo qem-dashboard was started (see https://confluence.suse.com/display/~coolo/Maintenance+CI+Dashboard?focusedCommentId=876905006#comment-876905006) to provide a database that provides track history. According to coolo "All code is between qa-tools members". I consider filtering emails is easier than being able to guess what could possibly block updates when reading IBS comments. coolo stated "https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.11/maillist.html is such a joyful read", likely meaning that the mailing list has high-volume and is hard to read, maybe made worse by the automatic commenting we introduced. However I consider the ML already hard to fully read as even without any automatic comments there are many emails, e.g. "Request XYZ requires review" and similar. So I think automatic comments when tests block release requests are not making the situation worse. Either people manage the email workload with proper processes and filtering or they ignore the emails or unsubscribe. We have other mailing lists which work the same. I stated "And all these 24 individuals and subscribers of the ML are welcome to unsubscribe or filter" and coolo answered "you can't unsubscribe from IBS comments!" which I don't know if that is true but in which case I consider filtering the right approach. coolo: "unless you unsubscribe from all of IBS email, which is not what I want - because I want to be notified about meaningful IBS comments. So you're asking 24 individuals to filter email that shouldn't have been sent to begin with? I review at times - like everyone else. Noone of the receivers reviews full time. But you also send it to all of maint-coord, autobuild and SLE release coord - and those are unreachable by IBS comments now because they did what you recommend: filter all IBS emails to /dev/null. Very few of them will even have bothered to filter the bot comments. And the email you send are even sent as coolo@suse.de, which makes me very valid for objection".
coolo: "Everyone is asking for smelt to show these comments - why not create them in smelt?". okurz: "for two reasons: 1. Because the information is also useful in IBS, 2. because my assumption was that it's easier to first rollback (or bring back) the previous feature before developing something new. Point 2 by now is questionable as it took 3 months (!) to achieve that however that does not mean that it would have been faster to do that in smelt. And that still leaves 1."
coolo: "Why it's useful there? Noone missed it there - or can you point to something I missed? What I would find useful instead: if the bot commented the moment it approved*. Because that's an IBS action - the review history is made for that".
okurz: "there are often questions like: "Why is SR#42 not approved by qam-openqa"? And that comment in IBS would directly give that answer. Alerting 24+ people that the bot looked at something - not so useful". okurz "well, if there would be a comment like "bot: I will not approve because tests failed, take a look on dashboard-url", that would be equivalent, yes. I agree that there are many points to improve but there is one more thing that comments plus the according email provide: A timely trigger. The 'look at the dashboard'-suggestion is good to see the current state at all times but IMHO not the best answer to everything. I also receive many other emails by OBS, IBS, github and most of them I use as a notifcation or a trigger point. I don't need to get all the information from email but at least the 'something happened' event.". coolo: "That's the reason the comment functionality was implemented for openSUSE. we had like 3 updates in the queue and got around 10 emails a day - everytime a trigger to go to openqa and check (and retrigger). but with SLE and IBS all that's left for you is: filter the mess into a folder you have on short delete expire. And the comment isn't really state of the art either". okurz: "Yes, everybody can choose that but what would be the answer to people that look for updates in a specific release request when there would be no emails? Poll a webUI manually? The comment could say 'I finished testing and I could not approve. For the current state go to http://...'.". coolo: "Again: I'm fine with having one comment once it decided it's not good enough. Just like I had initially in mind that the bot comments the final state when it does approve (e.g. the snapshot it used to approve it, but also the exceptions applied). I'm ok with having an IBS comment in 2 situations: the first time all tests finished (pointing to the dashboard only) and the moment the review is approved (with the current summary). So later on it can be verified what the bot actually looked at. Because once it approved the request, it's 'too late' and the machinery is rolling. If it turns out, the bot was buggy or didn't test product X we have to know that in the post mortem"
okurz: "But we want comments in the case of "there is at least one failed" test that blocks auto-approval. Would it help if we somehow say 'as soon as there is one comment on a release request we will never write one again'? I also told every QE squad that they can look at the mailing list same as the dashboard to look for tests that concern them"

Next steps

So I suggest the next step that we try is to only write a single comment per release request whenever either a first blocking failure was encountered or all currently scheduled tests for one release request have finished testing and there is at least one blocking failure. Also in the comment point to the current test results, e.g. something like an https://openqa.suse.de/tests/overview link

#23 Updated by mgrifalconi 4 months ago

No objections on the next steps proposed from my side.

I could also see this be kept on the side if
https://progress.opensuse.org/issues/97118 #97118
https://progress.opensuse.org/issues/104209 #104209
https://progress.opensuse.org/issues/97274 #97274

Will get more priority. The important thing IMHO is to at least have 'something' that allows a review of past days and not only the latest one, no matter the implementation.

#24 Updated by cdywan 3 months ago

Being an epic, this ticket has no priority, but existing subtasks were High and we should define new ones at the next opportunity

#25 Updated by okurz 3 months ago

12901

It seems I have new information. Apparently one can customize if or what notifications are sent. On the page https://build.suse.de/my/subscriptions I found

Screenshot_20220222_102657_ibs_notifictions_qam-openqa_configurable_email_web.png

which looks like the exact thing that coolo stated wasn't possible. Well, maybe it is just possible now. So with this I suggest we simply enable the old comments as was implemented and everybody can unsubscribe for the specific group "qam-openqa" based on their individual preferences. Any objections? If not I will ask for the comments to be enabled again after a waiting time of 14 days, that is 2022-03-10.

#26 Updated by cdywan about 1 month ago

okurz wrote:

Any objections? If not I will ask for the comments to be enabled again after a waiting time of 14 days, that is 2022-03-10.

So do we want to proceed here? Or has this already been done? Time's up.

#27 Updated by okurz about 1 month ago

  • Subject changed from [epic] enable qa-maintenance/openQABot comments on smelt again to [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)
  • Status changed from Feedback to Workable
  • Assignee changed from okurz to jbaier_cz

Discussed in weekly SUSE QE Tools meeting 2022-04-08. I will announce in some Slack channels that we will enable comments again.

jbaier_cz as discussed please add the according gitlab CI job to enable comments again.

#28 Updated by okurz about 1 month ago

  • Copied to action #109701: enable qem-bot comments on IBS again after subscriptions can be personally configured added

#29 Updated by okurz about 1 month ago

  • Status changed from Workable to Feedback
  • Assignee changed from jbaier_cz to okurz

Announced in

jbaier_cz I moved the "enable again" specific task to #109701

#30 Updated by coolo about 1 month ago

Comment #97121#note-23 asks for "some way" not comments - and the dashboard implements that way now. So what's the point in keeping with the plan to spam the masses?

#31 Updated by okurz about 1 month ago

coolo wrote:

Comment #97121#note-23 asks for "some way" not comments - and the dashboard implements that way now.

Can you be more specific about what you mean by that? What can the dashboard do?

So what's the point in keeping with the plan to spam the masses?

The point is that we are not progressing otherwise, nobody added test coverage, nobody dares to add a lot of features to qem-bot nor qem-dashboard, people still ask for notifications (not dashboards that need to be polled) and OBS has configurable notifications.

EDIT: Also it was mentioned in https://suse.slack.com/archives/C02CCRM8946/p1649413183546269 "these comments go to all of maintenance team and autobuild team and sle release coordinators as well, not just [qam-openqa], who's members may have good reason to get notifications for all kind of IBS things"

#32 Updated by okurz about 1 month ago

I disabled the comments again in https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules/115/edit after seeing comments from this morning. The feature works not at all as one should expect. For example https://build.suse.de/request/show/268912#comment-3859441 shows

Group Containers Maintenance Updates@Server-DVD-Updates (22 tests failed)
sle-15-SP3-Server-DVD-Updates-aarch64-Build20220409-1-podman_tests@aarch64-virtio is waiting
sle-15-SP3-Server-DVD-Updates-aarch64-Build20220409-1-docker_tests@aarch64-virtio is waiting

Following the first link I can find 8 scheduled jobs on https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1 not 22 failing tests. And there is a huge list of "is waiting" tests that are not even completed yet. Why would we report about that? There should be reports about requests which are not auto-approved because there are tests which failed but I can't find such tests in the above example. And it might be helpful to report that besides any failed tests there are also some which are not even completed at the time of writing but not a long list for each individual job. We should rediscuss expectations and options before continuing.

#33 Updated by coolo about 1 month ago

And compare to https://dashboard.qam.suse.de/incident/23521 giving you a live view on the past days and why they failed for the reviewer to judge if these failures were related to mozilla-nss... which is what #23 asked for.

#34 Updated by jbaier_cz about 1 month ago

okurz wrote:

Following the first link I can find 8 scheduled jobs on https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1 not 22 failing tests.

However there were exactly 22 "is waiting" tests. I would guess, this is because the state is not recognized in https://github.com/openSUSE/qem-bot/blob/master/openqabot/commenter.py#L51 so the bot assumes it is a failure.

#35 Updated by okurz about 1 month ago

  • Parent task set to #99303

coolo wrote:

And compare to https://dashboard.qam.suse.de/incident/23521 giving you a live view on the past days and why they failed for the reviewer to judge if these failures were related to mozilla-nss... which is what #23 asked for.

Yes, you stated already that in comment 23 - reference as #97121#note-23, not #23 – the dashboard would have implemented something. Can you please explain that in more detail? I don't know what you mean.

jbaier_cz wrote:

okurz wrote:

Following the first link I can find 8 scheduled jobs on https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1 not 22 failing tests.

However there were exactly 22 "is waiting" tests. I would guess, this is because the state is not recognized in https://github.com/openSUSE/qem-bot/blob/master/openqabot/commenter.py#L51 so the bot assumes it is a failure.

Maybe. But it shows that the situation is worse than I expected. Also the main problem I see is that there should have only been a comment at all if the auto-approval is blocked which at that point was not the case because there was not a single real failure that would block auto-approval. We would have needed to wait for jobs to complete before making such call.

#36 Updated by okurz 15 days ago

  • Description updated (diff)
  • Status changed from Feedback to Workable

need to define subtasks.

Also available in: Atom PDF