Project

General

Profile

coordination #97121

[epic] enable qa-maintenance/openQABot comments on smelt again

Added by mgrifalconi 2 months ago. Updated 27 days ago.

Status:
Blocked
Priority:
High
Assignee:
Target version:
Start date:
2021-09-14
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)

Description

While the qem dashboard is agreed to not yet be productive, smelt comments are very useful while doing manual approval of updates.

I would like to have such resource back, at least until https://progress.opensuse.org/issues/97118 #97118 is sorted out.


Subtasks

action #98637: [timeboxed:20h] try to enable comments on IBS (and smelt) again from SUSE QA maintenance openQA test results size:MWorkable


Related issues

Related to QA - action #96998: Increase bus factor for bot-ng size:MResolved2021-07-08

History

#1 Updated by mgrifalconi 2 months ago

  • Priority changed from Normal to Urgent

Raised priority since review becomes more difficult without this feature

#2 Updated by okurz 2 months ago

  • Project changed from openQA Project to QA
  • Subject changed from enable bot comments on smelt again to enable qa-maintenance/openQABot comments on smelt again
  • Target version set to Ready

#3 Updated by vpelcak 2 months ago

Hello.

openQA bot was commenting in IBS which lead to massive spam of e-mails.
Besides we also saw opportunities in CI dashboard for the openQA review:
https://progress.opensuse.org/issues/80194 #80194

Therefore some 2 weeks ago we had a meeting - openQA reviewers, Ondrej and me, where openQA reviewers agreed to switch from openQA bot to CI dashboard.
The loss of comments was discussed there as well.

Further usability improvements could be discussed with Sebastian Riedel.

#4 Updated by okurz 2 months ago

  • Description updated (diff)

#5 Updated by okurz 2 months ago

  • Related to action #96998: Increase bus factor for bot-ng size:M added

#6 Updated by okurz 2 months ago

  • Assignee set to okurz

I will clarify before moving on asking the team to conduct work here.

#7 Updated by okurz 2 months ago

  • Status changed from New to Feedback

vpelcak wrote:

Hello.

openQA bot was commenting in IBS which lead to massive spam of e-mails.
Besides we also saw opportunities in CI dashboard for the openQA review:
https://progress.opensuse.org/issues/80194 #80194

Therefore some 2 weeks ago we had a meeting - openQA reviewers, Ondrej and me, where openQA reviewers agreed to switch from openQA bot to CI dashboard.

In particular for the point "openQA reviewers, Ondrej and me […] agreed to switch from openQA bot to CI dashboard" this seems to be in conflict with what you expect from me as product owner for SUSE QE Tools and also what you and others expect from the team SUSE QE Tools. I see a problem not only with the technical impacts described above but also with how such a decision was made. One more reason why I am wondering about such a decision which steps would need to be conducted for any "QAM dashboard" first before continuing such changes which impact the process.

The loss of comments was discussed there as well.

And what are the results of the discussion? I am particularly interested as you know that some people still would like to have "notifications".

For SUSE QE maintenance tests we already have a mailing list qa-maintenance-reports@suse.de that receives comments from https://gitlab.suse.de/qa-maintenance/openQABot/ on build.suse.de whenever any incident related tests fail. The archive can be seen on
http://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/index.html . https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.08/maillist.html shows that the last email with "Request … commented by sle-qam-openqa" was from 2021-08-07. I would like to know who applied the change and where the change was applied. Can you clarify that point?

#9 Updated by okurz 2 months ago

So if I understand correctly osukup disabled the schedules for "full" and "incident" tests and instead scheduled them on his workstation manually using https://gitlab.suse.de/qa-maintenance/bot-ng, a personal internal prototype. I would prefer to run "full" tests on qa-maintenance/openQABot again until qa-maintenance/bot-ng is production ready with team responsibility at least after #96998 before we adjust the schedule again. I will ask vpelcak and osukup about the feasibility.

#10 Updated by okurz about 2 months ago

  • Due date set to 2021-09-13
  • Priority changed from Urgent to Normal

There were no objections from the side of vpelcak and no comment from osukup. In the meantime with #96998 resolved at least the bus factor for bot-ng increased. I have asked lask week in https://chat.suse.de/channel/qem-openqa-review?msg=vuBWpiRhgsJRGqwwj and there was no clear positive nor negative feedback. So I assume we can deprioritize.

Open points I like to clarify:

  1. What is the position of mgrifalconi (as he created the ticket initially)
  2. how are reports of openQA tests at the time of acceptance provided if not by the comments on IBS/smelt
  3. how can we send notification emails when there are new test results regarding incidents

#11 Updated by VANASTASIADIS about 2 months ago

I agree with okurz on this, there were complaints about the loss of comments from other sides as well, as this leads to the gear icons in smelt not pointing to the existing tests.

Can we discuss about restoring that functionality? If this info exists but in another place (i.e. the dashboard) we should at least provide the link on smelt to make testers aware.

#12 Updated by bschmidt about 2 months ago

I'd like to chime in here. The link from https://maintenance.suse.de/ to the corresponding test in openQA was quite convenient as long as it where there. So, please restore or replace the now missing functionality.
Just for reference:
https://suse.slack.com/archives/C02D12FF0H1/p1631087431032900

#13 Updated by osukup about 2 months ago

maybe the acceptable solution for this feature request will be https://gitlab.suse.de/opensuse/qem-dashboard/-/issues/15 and then use information from qem-dashboard in SMELT

#15 Updated by okurz about 1 month ago

  • Status changed from Feedback to Workable
  • Assignee deleted (okurz)
  • Priority changed from Normal to High

Seeing the feedback from multiple persons I see multiple problems with the approach that was taken which I see as one more reason to go back to the old approach. I simply do not see the "QEM dashboard" ready yet for production and IMHO the impact of relying on it has not been fully envisioned.

As there were no objections from vpelcak (see #97121#note-10) I suggest again to go back to the old ways for now. Putting back to "Workable" for the team to pick up.

#17 Updated by okurz about 1 month ago

Discussed this with osukup, VANASTASIADIS, jbaier_cz and I see the following three alternatives:

  1. Go back to openQABot which is currently the only component supporting commenting on IBS. According to osukup this would need changes again in code and "metadata" for public cloud as some changes have been done there but not synced back
  2. Go forward with https://gitlab.suse.de/tools/smelt/-/issues/724, i.e. reading out the data that bot-ng puts into qem dashboard. What is certainly not so nice is that the decision to switch to bot-ng and not have IBS comments broke existing workflows and now pressuring smelt developers to use the data from the new source.
  3. Implementing IBS commenting into bot-ng. This would bring back comments (which of course can again annoy some users that don't want them) but still allows smelt to read the data

Regarding the data in qem dashboard I understood that in case the database would be lost likely bot-ng can push all relevant data into an empty database again. The only bad impact in that case would be that potentially unnecessary reschedules of openQA tests would happen which should be acceptable in case of disaster recovery scenarios.

With this I suggest to go for alternative 3., implementing IBS commenting into bot-ng, with a switch to be able to activate/deactivate. What are the opinions of others? Please state your opinions.

#18 Updated by mgrifalconi about 1 month ago

Hello,

While I still question the hard switch to a new tool that do not cover all the features of the old one, I agree on moving forward instead of rollback.
I have no preference on where I can see the data SMELT or dashboard, as long as it is there (or even better, the bot would be able to process in my place, see https://progress.opensuse.org/issues/97274 #97274).

I would be curious to hear from other stakeholders as well, like other reviewers.

#19 Updated by okurz about 1 month ago

Regarding the proposed solution nr 2. I wonder how notifications can be triggered as soon as tests blocking auto-approval fail as I don't want to rely on humans periodically polling dashboards. Enabling comments sent again would cover that point but maybe I am missing something and there is a better way of pushing the information event-based? Also asked in https://suse.slack.com/archives/C02D16TCP99/p1631607307044800

#20 Updated by cdywan about 1 month ago

I wonder if it would be sensible for this ticket to block on #98601 - or alternatively Feedback with an updated Due Date and some idea how to come up with clear ACs. To me it doesn't look "Workable" right now.

#21 Updated by okurz about 1 month ago

  • Tracker changed from action to coordination
  • Subject changed from enable qa-maintenance/openQABot comments on smelt again to [epic] enable qa-maintenance/openQABot comments on smelt again
  • Status changed from Workable to Blocked
  • Assignee set to okurz

cdywan wrote:

I wonder if it would be sensible for this ticket to block on #98601

no, because this ticket is about "bring back old behaviour", which does not need any more detailed specs.

  • or alternatively Feedback with an updated Due Date and some idea how to come up with clear ACs. To me it doesn't look "Workable" right now.

Right, we did not estimate as such. To make it more clear I now created an urgent subtask #98637 for the specific task to enable IBS comments again, same as we had in before, and let this ticket stay as an epic which can be used to catch more high-level feedback and to answer additional open points.

Also available in: Atom PDF