https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-08-18T12:50:42ZopenSUSE Project Management ToolQA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4363552021-08-18T12:50:42Zmgrifalconi
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li></ul><p>Raised priority since review becomes more difficult without this feature</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4363672021-08-18T13:16:17Zokurzokurz@suse.com
<ul><li><strong>Project</strong> changed from <i>openQA Project</i> to <i>QA</i></li><li><strong>Subject</strong> changed from <i>enable bot comments on smelt again</i> to <i>enable qa-maintenance/openQABot comments on smelt again</i></li><li><strong>Target version</strong> set to <i>Ready</i></li></ul> QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4364212021-08-18T15:06:19Zvpelcakvpelcak@suse.com
<ul></ul><p>Hello.</p>
<p>openQA bot was commenting in IBS which lead to massive spam of e-mails.<br>
Besides we also saw opportunities in CI dashboard for the openQA review:<br>
<a href="https://progress.opensuse.org/issues/80194" class="external">https://progress.opensuse.org/issues/80194</a> #80194</p>
<p>Therefore some 2 weeks ago we had a meeting - openQA reviewers, Ondrej and me, where openQA reviewers agreed to switch from openQA bot to CI dashboard.<br>
The loss of comments was discussed there as well.</p>
<p>Further usability improvements could be discussed with Sebastian Riedel.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4374592021-08-23T09:19:33Zokurzokurz@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/437459/diff?detail_id=414782">diff</a>)</li></ul> QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4374712021-08-23T09:22:03Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/96998">action #96998</a>: Increase bus factor for bot-ng size:M</i> added</li></ul> QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4374772021-08-23T09:24:17Zokurzokurz@suse.com
<ul><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>I will clarify before moving on asking the team to conduct work here.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4375522021-08-23T14:49:48Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li></ul><p>vpelcak wrote:</p>
<blockquote>
<p>Hello.</p>
<p>openQA bot was commenting in IBS which lead to massive spam of e-mails.<br>
Besides we also saw opportunities in CI dashboard for the openQA review:<br>
<a href="https://progress.opensuse.org/issues/80194" class="external">https://progress.opensuse.org/issues/80194</a> #80194</p>
<p>Therefore some 2 weeks ago we had a meeting - openQA reviewers, Ondrej and me, where openQA reviewers agreed to switch from openQA bot to CI dashboard.</p>
</blockquote>
<p>In particular for the point "openQA reviewers, Ondrej and me […] agreed to switch from openQA bot to CI dashboard" this seems to be in conflict with what you expect from me as product owner for SUSE QE Tools and also what you and others expect from the team SUSE QE Tools. I see a problem not only with the technical impacts described above but also with how such a decision was made. One more reason why I am wondering about such a decision which steps would need to be conducted for any "QAM dashboard" first before continuing such changes which impact the process.</p>
<blockquote>
<p>The loss of comments was discussed there as well.</p>
</blockquote>
<p>And what are the results of the discussion? I am particularly interested as you know that some people still would like to have "notifications".</p>
<p>For SUSE QE maintenance tests we already have a mailing list <a href="mailto:qa-maintenance-reports@suse.de">qa-maintenance-reports@suse.de</a> that receives comments from <a href="https://gitlab.suse.de/qa-maintenance/openQABot/" class="external">https://gitlab.suse.de/qa-maintenance/openQABot/</a> on build.suse.de whenever any incident related tests fail. The archive can be seen on<br>
<a href="http://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/index.html" class="external">http://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/index.html</a> . <a href="https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.08/maillist.html" class="external">https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.08/maillist.html</a> shows that the last email with "Request … commented by sle-qam-openqa" was from 2021-08-07. I would like to know who applied the change and where the change was applied. Can you clarify that point?</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4387822021-08-26T13:52:36Zokurzokurz@suse.com
<ul></ul><p>So if I understand correctly osukup disabled the schedules for "full" and "incident" tests and instead scheduled them on his workstation manually using <a href="https://gitlab.suse.de/qa-maintenance/bot-ng" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng</a>, a personal internal prototype. I would prefer to run "full" tests on qa-maintenance/openQABot again until qa-maintenance/bot-ng is production ready with team responsibility at least after <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: Increase bus factor for bot-ng size:M (Resolved)" href="https://progress.opensuse.org/issues/96998">#96998</a> before we adjust the schedule again. I will ask vpelcak and osukup about the feasibility.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4400932021-09-01T06:56:20Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> set to <i>2021-09-13</i></li><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Normal</i></li></ul><p>There were no objections from the side of vpelcak and no comment from osukup. In the meantime with <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: Increase bus factor for bot-ng size:M (Resolved)" href="https://progress.opensuse.org/issues/96998">#96998</a> resolved at least the bus factor for bot-ng increased. I have asked lask week in <a href="https://chat.suse.de/channel/qem-openqa-review?msg=vuBWpiRhgsJRGqwwj" class="external">https://chat.suse.de/channel/qem-openqa-review?msg=vuBWpiRhgsJRGqwwj</a> and there was no clear positive nor negative feedback. So I assume we can deprioritize.</p>
<p>Open points I like to clarify:</p>
<ol>
<li>What is the position of mgrifalconi (as he created the ticket initially)</li>
<li>how are reports of openQA tests at the time of acceptance provided if not by the comments on IBS/smelt</li>
<li>how can we send notification emails when there are new test results regarding incidents</li>
</ol>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4433072021-09-08T08:22:30ZVANASTASIADISvasilios.anastasiadis@suse.com
<ul></ul><p>I agree with okurz on this, there were complaints about the loss of comments from other sides as well, as this leads to the gear icons in smelt not pointing to the existing tests. </p>
<p>Can we discuss about restoring that functionality? If this info exists but in another place (i.e. the dashboard) we should at least provide the link on smelt to make testers aware.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4434062021-09-08T08:33:52Zbschmidtbschmidt@suse.com
<ul></ul><p>I'd like to chime in here. The link from <a href="https://maintenance.suse.de/" class="external">https://maintenance.suse.de/</a> to the corresponding test in openQA was quite convenient as long as it where there. So, please restore or replace the now missing functionality. <br>
Just for reference:<br>
<a href="https://suse.slack.com/archives/C02D12FF0H1/p1631087431032900" class="external">https://suse.slack.com/archives/C02D12FF0H1/p1631087431032900</a></p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4434362021-09-08T08:51:52Zosukup
<ul></ul><p>maybe the acceptable solution for this feature request will be <a href="https://gitlab.suse.de/opensuse/qem-dashboard/-/issues/15" class="external">https://gitlab.suse.de/opensuse/qem-dashboard/-/issues/15</a> and then use information from qem-dashboard in SMELT</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4449482021-09-11T19:59:22Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Workable</i></li><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul><p>Seeing the feedback from multiple persons I see multiple problems with the approach that was taken which I see as one more reason to go back to the old approach. I simply do not see the "QEM dashboard" ready yet for production and IMHO the impact of relying on it has not been fully envisioned.</p>
<p>As there were no objections from vpelcak (see <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121#note-10">#97121#note-10</a>) I suggest again to go back to the old ways for now. Putting back to "Workable" for the team to pick up.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4449542021-09-12T08:29:24Zosukup
<ul></ul><p>--> <a href="https://gitlab.suse.de/tools/smelt/-/issues/724" class="external">https://gitlab.suse.de/tools/smelt/-/issues/724</a></p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4454522021-09-13T20:30:46Zokurzokurz@suse.com
<ul></ul><p>Discussed this with osukup, VANASTASIADIS, jbaier_cz and I see the following three alternatives:</p>
<ol>
<li>Go back to openQABot which is currently the only component supporting commenting on IBS. According to osukup this would need changes again in code and "metadata" for public cloud as some changes have been done there but not synced back</li>
<li>Go forward with <a href="https://gitlab.suse.de/tools/smelt/-/issues/724" class="external">https://gitlab.suse.de/tools/smelt/-/issues/724</a>, i.e. reading out the data that bot-ng puts into qem dashboard. What is certainly not so nice is that the decision to switch to bot-ng and not have IBS comments broke existing workflows and now pressuring smelt developers to use the data from the new source.</li>
<li>Implementing IBS commenting into bot-ng. This would bring back comments (which of course can again annoy some users that don't want them) but still allows smelt to read the data</li>
</ol>
<p>Regarding the data in qem dashboard I understood that in case the database would be lost likely bot-ng can push all relevant data into an empty database again. The only bad impact in that case would be that potentially unnecessary reschedules of openQA tests would happen which should be acceptable in case of disaster recovery scenarios.</p>
<p>With this I suggest to go for alternative 3., implementing IBS commenting into bot-ng, with a switch to be able to activate/deactivate. What are the opinions of others? Please state your opinions.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4455662021-09-14T07:17:22Zmgrifalconi
<ul></ul><p>Hello, </p>
<p>While I still question the hard switch to a new tool that do not cover all the features of the old one, I agree on moving forward instead of rollback.<br>
I have no preference on where I can see the data SMELT or dashboard, as long as it is there (or even better, the bot would be able to process in my place, see <a href="https://progress.opensuse.org/issues/97274" class="external">https://progress.opensuse.org/issues/97274</a> <a class="issue tracker-4 status-1 priority-4 priority-default child parent" title="action: qam dashboard improvement ideas (New)" href="https://progress.opensuse.org/issues/97274">#97274</a>).</p>
<p>I would be curious to hear from other stakeholders as well, like other reviewers.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4455992021-09-14T08:13:04Zokurzokurz@suse.com
<ul></ul><p>Regarding the proposed solution nr 2. I wonder how notifications can be triggered as soon as tests blocking auto-approval fail as I don't want to rely on humans periodically polling dashboards. Enabling comments sent again would cover that point but maybe I am missing something and there is a better way of pushing the information event-based? Also asked in <a href="https://suse.slack.com/archives/C02D16TCP99/p1631607307044800" class="external">https://suse.slack.com/archives/C02D16TCP99/p1631607307044800</a></p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4456922021-09-14T12:31:07Zlivdywanliv.dywan@suse.com
<ul></ul><p>I wonder if it would be sensible for this ticket to block on #98601 - or alternatively <em>Feedback</em> with an updated <em>Due Date</em> and some idea how to come up with clear ACs. To me it doesn't look "Workable" right now.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4458002021-09-14T19:31:17Zokurzokurz@suse.com
<ul><li><strong>Tracker</strong> changed from <i>action</i> to <i>coordination</i></li><li><strong>Subject</strong> changed from <i>enable qa-maintenance/openQABot comments on smelt again</i> to <i>[epic] enable qa-maintenance/openQABot comments on smelt again</i></li><li><strong>Status</strong> changed from <i>Workable</i> to <i>Blocked</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>cdywan wrote:</p>
<blockquote>
<p>I wonder if it would be sensible for this ticket to block on #98601</p>
</blockquote>
<p>no, because this ticket is about "bring back old behaviour", which does not need any more detailed specs.</p>
<blockquote>
<ul>
<li>or alternatively <em>Feedback</em> with an updated <em>Due Date</em> and some idea how to come up with clear ACs. To me it doesn't look "Workable" right now.</li>
</ul>
</blockquote>
<p>Right, we did not estimate as such. To make it more clear I now created an urgent subtask <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [timeboxed:20h] try to enable comments on IBS (and smelt) again from SUSE QA maintenance openQA t... (Resolved)" href="https://progress.opensuse.org/issues/98637">#98637</a> for the specific task to enable IBS comments again, same as we had in before, and let this ticket stay as an epic which can be used to catch more high-level feedback and to answer additional open points.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4672832021-11-24T07:50:55Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Blocked</i> to <i>Feedback</i></li></ul><a name="Current-state"></a>
<h2 >Current state<a href="#Current-state" class="wiki-anchor">¶</a></h2>
<p>With <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [timeboxed:20h] try to enable comments on IBS (and smelt) again from SUSE QA maintenance openQA t... (Resolved)" href="https://progress.opensuse.org/issues/98637">#98637</a> resolved we have a feature that allows to send IBS comments and hence emails whenever release requests are blocked from auto-approval due to failing tests. This is an important goal reached. However we did not yet enable this feature to automatically and recurringly send comments yet due to objections.</p>
<a name="Discussion"></a>
<h2 >Discussion<a href="#Discussion" class="wiki-anchor">¶</a></h2>
<p>There was a lengthy discussion in <a href="https://suse.slack.com/archives/C02CANHLANP/p1637593271430900">https://suse.slack.com/archives/C02CANHLANP/p1637593271430900</a> (internal). I am trying to note the important points. Whenever someone writes a comment on IBS that sends emails. This is in general something which I want to have as notification feature but when someone is subscribed to high-traffic projects, many packages, etc., there can be very many emails, referenced as "spam" by some, e.g. coolo, hence a <em>problem</em>. According to coolo this can account for roughly 1k emails per day which is of course a lot. According to coolo originally someone wanted to have IBS comments to "track history" or something. This is certainly not the best way and I don't consider this approach good for that purpose but for different purposes. So maintenance representatives, e.g. Stephan Barth, would like to see comments back on release requests so do I because the submitters learn that their release request is blocked by failing openQA tests. Resending more emails to just transport the same message can deteriorate quickly so maybe a good <em>idea</em> is to only ever send one comment (hence also one email) whenever failing tests block auto-approval and then it is up to submitters and QA engineers to resolve the situation. According to coolo qem-dashboard was started (see <a href="https://confluence.suse.com/display/%7Ecoolo/Maintenance+CI+Dashboard?focusedCommentId=876905006#comment-876905006">https://confluence.suse.com/display/~coolo/Maintenance+CI+Dashboard?focusedCommentId=876905006#comment-876905006</a>) to provide a database that provides track history. According to coolo "All code is between qa-tools members". I consider filtering emails is easier than being able to guess what could possibly block updates when reading IBS comments. coolo stated "<a href="https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.11/maillist.html">https://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/2021/qa-maintenance-reports.2021.11/maillist.html</a> is such a joyful read", likely meaning that the mailing list has high-volume and is hard to read, maybe made worse by the automatic commenting we introduced. However I consider the ML already hard to fully read as even without any automatic comments there are many emails, e.g. "Request XYZ requires review" and similar. So I think automatic comments when tests block release requests are not making the situation worse. Either people manage the email workload with proper processes and filtering or they ignore the emails or unsubscribe. We have other mailing lists which work the same. I stated "And all these 24 individuals and subscribers of the ML are welcome to unsubscribe or filter" and coolo answered "you can't unsubscribe from IBS comments!" which I don't know if that is true but in which case I consider filtering the right approach. coolo: "unless you unsubscribe from all of IBS email, which is not what I want - because I want to be notified about meaningful IBS comments. So you're asking 24 individuals to filter email that shouldn't have been sent to begin with? I review at times - like everyone else. Noone of the receivers reviews full time. But you also send it to all of maint-coord, autobuild and SLE release coord - and those are unreachable by IBS comments now because they did what you recommend: filter all IBS emails to /dev/null. Very few of them will even have bothered to filter the bot comments. And the email you send are even sent as <a href="mailto:coolo@suse.de">coolo@suse.de</a>, which makes me very valid for objection".<br>
coolo: "Everyone is asking for smelt to show these comments - why not create them in smelt?". okurz: "for two reasons: 1. Because the information is also useful in IBS, 2. because my assumption was that it's easier to first rollback (or bring back) the previous feature before developing something new. Point 2 by now is questionable as it took 3 months (!) to achieve that however that does not mean that it would have been faster to do that in smelt. And that still leaves 1."<br>
coolo: "Why it's useful there? Noone missed it there - or can you point to something I missed? What I would find useful instead: if the bot commented the moment it approved*. Because that's an IBS action - the review history is made for that".<br>
okurz: "there are often questions like: "Why is SR#42 not approved by qam-openqa"? And that comment in IBS would directly give that answer. Alerting 24+ people that the bot looked at something - not so useful". okurz "well, if there would be a comment like "bot: I will not approve because tests failed, take a look on dashboard-url", that would be equivalent, yes. I agree that there are many points to improve but there is one more thing that comments plus the according email provide: A timely trigger. The 'look at the dashboard'-suggestion is good to see the current state at all times but IMHO not the best answer to everything. I also receive many other emails by OBS, IBS, github and most of them I use as a notifcation or a trigger point. I don't need to get all the information from email but at least the 'something happened' event.". coolo: "That's the reason the comment functionality was implemented for openSUSE. we had like 3 updates in the queue and got around 10 emails a day - everytime a trigger to go to openqa and check (and retrigger). but with SLE and IBS all that's left for you is: filter the mess into a folder you have on short delete expire. And the comment isn't really state of the art either". okurz: "Yes, everybody can choose that but what would be the answer to people that look for updates in a specific release request when there would be no emails? Poll a webUI manually? The comment could say 'I finished testing and I could not approve. For the current state go to http://...'.". coolo: "Again: I'm fine with having one comment once it decided it's not good enough. Just like I had initially in mind that the bot comments the final state when it does approve (e.g. the snapshot it used to approve it, but also the exceptions applied). I'm ok with having an IBS comment in 2 situations: the first time all tests finished (pointing to the dashboard only) and the moment the review is approved (with the current summary). So later on it can be verified what the bot actually looked at. Because once it approved the request, it's 'too late' and the machinery is rolling. If it turns out, the bot was buggy or didn't test product X we have to know that in the post mortem"<br>
okurz: "But we want comments in the case of "there is at least one failed" test that blocks auto-approval. Would it help if we somehow say 'as soon as there is one comment on a release request we will never write one again'? I also told every QE squad that they can look at the mailing list same as the dashboard to look for tests that concern them"</p>
<a name="Next-steps"></a>
<h2 >Next steps<a href="#Next-steps" class="wiki-anchor">¶</a></h2>
<p>So I suggest the next step that we try is to only write a <em>single</em> comment per release request whenever either a first blocking failure was encountered <em>or</em> all currently scheduled tests for one release request have finished testing and there is at least one blocking failure. Also in the comment point to the current test results, e.g. something like an <a href="https://openqa.suse.de/tests/overview">https://openqa.suse.de/tests/overview</a> link</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4781312022-01-07T09:48:26Zmgrifalconi
<ul></ul><p>No objections on the next steps proposed from my side.</p>
<p>I could also see this be kept on the side if<br>
<a href="https://progress.opensuse.org/issues/97118" class="external">https://progress.opensuse.org/issues/97118</a> <a class="issue tracker-4 status-4 priority-4 priority-default child" title="action: enhance bot automatic approval: check multiple days (Feedback)" href="https://progress.opensuse.org/issues/97118">#97118</a><br>
<a href="https://progress.opensuse.org/issues/104209" class="external">https://progress.opensuse.org/issues/104209</a> <a class="issue tracker-4 status-6 priority-4 priority-default closed child" title="action: [qem] dashboard.qam.suse.de checkpoints for aggregates (Rejected)" href="https://progress.opensuse.org/issues/104209">#104209</a><br>
<a href="https://progress.opensuse.org/issues/97274" class="external">https://progress.opensuse.org/issues/97274</a> <a class="issue tracker-4 status-1 priority-4 priority-default child parent" title="action: qam dashboard improvement ideas (New)" href="https://progress.opensuse.org/issues/97274">#97274</a></p>
<p>Will get more priority. The important thing IMHO is to at least have 'something' that allows a review of past days and not only the latest one, no matter the implementation.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4946802022-02-23T22:36:04Zlivdywanliv.dywan@suse.com
<ul></ul><p>Being an epic, this ticket has no priority, but existing subtasks were High and we should define new ones at the next opportunity</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=4953912022-02-24T20:18:05Zokurzokurz@suse.com
<ul><li><strong>File</strong> <a href="/attachments/12901">Screenshot_20220222_102657_ibs_notifictions_qam-openqa_configurable_email_web.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/12901/Screenshot_20220222_102657_ibs_notifictions_qam-openqa_configurable_email_web.png">Screenshot_20220222_102657_ibs_notifictions_qam-openqa_configurable_email_web.png</a> added</li></ul><p>It seems I have new information. Apparently one <em>can</em> customize if or what notifications are sent. On the page <a href="https://build.suse.de/my/subscriptions" class="external">https://build.suse.de/my/subscriptions</a> I found</p>
<p><img src="https://progress.opensuse.org/attachments/download/12901/Screenshot_20220222_102657_ibs_notifictions_qam-openqa_configurable_email_web.png" alt="Screenshot_20220222_102657_ibs_notifictions_qam-openqa_configurable_email_web.png" loading="lazy" /></p>
<p>which looks like the exact thing that coolo stated wasn't possible. Well, maybe it is just possible now. So with this I suggest we simply enable the old comments as was implemented and everybody can unsubscribe for the specific group "qam-openqa" based on their individual preferences. Any objections? If not I will ask for the comments to be enabled again after a waiting time of 14 days, that is 2022-03-10.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5094692022-04-08T09:30:19Zlivdywanliv.dywan@suse.com
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p>Any objections? If not I will ask for the comments to be enabled again after a waiting time of 14 days, that is 2022-03-10.</p>
</blockquote>
<p>So do we want to proceed here? Or has this already been done? Time's up.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5095142022-04-08T10:14:14Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>[epic] enable qa-maintenance/openQABot comments on smelt again</i> to <i>[epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)</i></li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Workable</i></li><li><strong>Assignee</strong> changed from <i>okurz</i> to <i>jbaier_cz</i></li></ul><p>Discussed in weekly SUSE QE Tools meeting 2022-04-08. I will announce in some Slack channels that we will enable comments again.</p>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/29611">@jbaier_cz</a> as discussed please add the according gitlab CI job to enable comments again.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5095202022-04-08T10:21:17Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed child" href="/issues/109701">action #109701</a>: enable qem-bot comments on IBS again after subscriptions can be personally configured</i> added</li></ul> QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5095262022-04-08T10:21:43Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>Feedback</i></li><li><strong>Assignee</strong> changed from <i>jbaier_cz</i> to <i>okurz</i></li></ul><p>Announced in</p>
<ul>
<li><a href="https://suse.slack.com/archives/C02CCRM8946/p1649413066956119" class="external">#discuss-maintenance</a></li>
<li><a href="https://suse.slack.com/archives/C02D16TCP99/p1649413053090279" class="external">#team-lsg-qe-openqa-review</a></li>
<li><a href="https://suse.slack.com/archives/C02CANHLANP/p1649413040753849" class="external">#eng-testing</a></li>
<li><a href="http://mailman.suse.de/mlarch/SuSE/openqa/index.html" class="external">openqa@suse.de</a></li>
</ul>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/29611">@jbaier_cz</a> I moved the "enable again" specific task to <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: enable qem-bot comments on IBS again after subscriptions can be personally configured (Resolved)" href="https://progress.opensuse.org/issues/109701">#109701</a></p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5095292022-04-08T10:21:57Zcoolocoolo@suse.com
<ul></ul><p>Comment <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121#note-23">#97121#note-23</a> asks for "some way" not comments - and the dashboard implements that way now. So what's the point in keeping with the plan to spam the masses?</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5095412022-04-08T10:25:35Zokurzokurz@suse.com
<ul></ul><p>coolo wrote:</p>
<blockquote>
<p>Comment <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121#note-23">#97121#note-23</a> asks for "some way" not comments - and the dashboard implements that way now.</p>
</blockquote>
<p>Can you be more specific about what you mean by that? What can the dashboard do?</p>
<blockquote>
<p>So what's the point in keeping with the plan to spam the masses?</p>
</blockquote>
<p>The point is that we are not progressing otherwise, nobody added test coverage, nobody dares to add a lot of features to qem-bot nor qem-dashboard, people still ask for notifications (not dashboards that need to be polled) and OBS has configurable notifications.</p>
<p>EDIT: Also it was mentioned in <a href="https://suse.slack.com/archives/C02CCRM8946/p1649413183546269" class="external">https://suse.slack.com/archives/C02CCRM8946/p1649413183546269</a> "these comments go to all of maintenance team and autobuild team and sle release coordinators as well, not just [qam-openqa], who's members may have good reason to get notifications for all kind of IBS things"</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5097722022-04-09T07:07:36Zokurzokurz@suse.com
<ul></ul><p>I disabled the comments again in <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules/115/edit" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules/115/edit</a> after seeing comments from this morning. The feature works not at all as one should expect. For example <a href="https://build.suse.de/request/show/268912#comment-3859441" class="external">https://build.suse.de/request/show/268912#comment-3859441</a> shows</p>
<blockquote>
<p><a href="https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1" class="external">Group Containers Maintenance Updates@Server-DVD-Updates</a> (22 tests failed)<br>
<a href="https://openqa.suse.de/tests/8498667" class="external">sle-15-SP3-Server-DVD-Updates-aarch64-Build20220409-1-podman_tests@aarch64-virtio</a> is waiting<br>
<a href="https://openqa.suse.de/tests/8498668" class="external">sle-15-SP3-Server-DVD-Updates-aarch64-Build20220409-1-docker_tests@aarch64-virtio</a> is waiting<br>
…</p>
</blockquote>
<p>Following the first link I can find 8 scheduled jobs on <a href="https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1" class="external">https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1</a> not 22 failing tests. And there is a huge list of "is waiting" tests that are not even completed yet. Why would we report about that? There should be reports about requests which are not auto-approved because there are tests which failed but I can't find such tests in the above example. And it might be helpful to report that besides any failed tests there are also some which are not even completed at the time of writing but not a long list for each individual job. We should rediscuss expectations and options before continuing.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5097842022-04-09T08:37:41Zcoolocoolo@suse.com
<ul></ul><p>And compare to <a href="https://dashboard.qam.suse.de/incident/23521" class="external">https://dashboard.qam.suse.de/incident/23521</a> giving you a live view on the past days and why they failed for the reviewer to judge if these failures were related to mozilla-nss... which is what #23 asked for.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5097992022-04-09T10:08:07Zjbaier_czjbaier@suse.cz
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p>Following the first link I can find 8 scheduled jobs on <a href="https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1" class="external">https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1</a> not 22 failing tests.</p>
</blockquote>
<p>However there were exactly 22 "is waiting" tests. I would guess, this is because the state is not recognized in <a href="https://github.com/openSUSE/qem-bot/blob/master/openqabot/commenter.py#L51" class="external">https://github.com/openSUSE/qem-bot/blob/master/openqabot/commenter.py#L51</a> so the bot assumes it is a failure.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5099012022-04-09T12:59:34Zokurzokurz@suse.com
<ul><li><strong>Parent task</strong> set to <i>#99303</i></li></ul><p>coolo wrote:</p>
<blockquote>
<p>And compare to <a href="https://dashboard.qam.suse.de/incident/23521" class="external">https://dashboard.qam.suse.de/incident/23521</a> giving you a live view on the past days and why they failed for the reviewer to judge if these failures were related to mozilla-nss... which is what #23 asked for.</p>
</blockquote>
<p>Yes, you stated already that in comment 23 - reference as <code>#97121#note-23</code>, not <code>#23</code> – the dashboard would have implemented something. Can you please explain that in more detail? I don't know what you mean.</p>
<p>jbaier_cz wrote:</p>
<blockquote>
<p>okurz wrote:</p>
<blockquote>
<p>Following the first link I can find 8 scheduled jobs on <a href="https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1" class="external">https://openqa.suse.de/tests/overview?version=15-SP3&groupid=417&flavor=Server-DVD-Updates&distri=sle&build=20220409-1</a> not 22 failing tests.</p>
</blockquote>
<p>However there were exactly 22 "is waiting" tests. I would guess, this is because the state is not recognized in <a href="https://github.com/openSUSE/qem-bot/blob/master/openqabot/commenter.py#L51" class="external">https://github.com/openSUSE/qem-bot/blob/master/openqabot/commenter.py#L51</a> so the bot assumes it is a failure.</p>
</blockquote>
<p>Maybe. But it shows that the situation is worse than I expected. Also the main problem I see is that there should have only been a comment at all if the auto-approval is blocked which at that point was not the case because there was not a single real failure that would block auto-approval. We would have needed to wait for jobs to complete before making such call.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5157882022-05-04T09:52:03Zokurzokurz@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/515788/diff?detail_id=487744">diff</a>)</li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Workable</i></li></ul><p>need to define subtasks.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5251852022-06-03T11:58:10Zlivdywanliv.dywan@suse.com
<ul></ul><p>okurz wrote:</p>
<blockquote>
<p>need to define subtasks.</p>
</blockquote>
<p>Reminder to self, we should really do this next week</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5252242022-06-03T14:42:05Zokurzokurz@suse.com
<ul><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li><li><strong>Target version</strong> changed from <i>Ready</i> to <i>future</i></li></ul><p>Waiting first for <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: qem-bot: Ignore not-ok openQA jobs for specific incident based on openQA job comment size:M (Resolved)" href="https://progress.opensuse.org/issues/107923">#107923</a> which might have a very slim chance of being related. Also I am actually moving the epic out of our backlog again for now. We have more pressing issues and other sagas to finish. The parent epic for this ticket <a class="issue tracker-6 status-1 priority-4 priority-default overdue parent behind-schedule" title="coordination: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, ... (New)" href="https://progress.opensuse.org/issues/99303">#99303</a> isn't within our current backlog anyway.</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5252272022-06-03T14:42:14Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>New</i></li></ul> QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=5390772022-07-20T10:45:50Zokurzokurz@suse.com
<ul></ul><p>By now the comments have been disabled again in <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules</a>. The problems with the comments are:</p>
<ul>
<li>Comments are written at a fixed time, not when tests fail or all tests finished</li>
<li>Comments are repeated every day, not updated</li>
<li>Comments are written for scheduled or still running tests, not waiting for tests to be finished</li>
</ul>
<p>I will create a new ticket with specific requirements</p>
QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=7491632024-01-04T12:57:16Zokurzokurz@suse.com
<ul><li><strong>Subtask</strong> <i>#153107</i> added</li></ul> QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=7572552024-01-29T17:33:43Zokurzokurz@suse.com
<ul><li><strong>Subtask</strong> <i>#154498</i> added</li></ul> QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=7588332024-02-01T14:35:56Zokurzokurz@suse.com
<ul><li><strong>Subtask</strong> <i>#154762</i> added</li></ul> QA - coordination #97121: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again)https://progress.opensuse.org/issues/97121?journal_id=7799562024-03-22T10:23:10Zokurzokurz@suse.com
<ul><li><strong>Subtask</strong> <i>#157741</i> added</li></ul>