QA (public)

@okurz I understand that tools team is busy, but be would be possible to increase priority from low to something higher? This dashboard visibility issue makes regular confusion about maintenance update status.

Hint from Oliver (https://suse.slack.com/archives/C02DQJKULE4/p1722327940699429?thread_ts=1722235479.858789&cid=C02DQJKULE4):

https://github.com/openSUSE/qem-bot/blob/463d5de225767791d0b6815435a0b341ce319c67/openqabot/approver.py#L129 is where the "acceptable_for" comment in qem-bot is parsed and evaluated. The challenge is that this is called during the "approval" command which is not syncing anything to the dashboard (yet).

I guess we could sync whether each job is acceptable for what incidents like this: https://github.com/openSUSE/qem-bot/pull/190

This has the advantage of doing this kind of updating in the "sync-incidents" commands which seems to follow how this bot is designed.

However, it has the disadvantage that the dashboard needs to compute whether an incident is approvable or not based on this information we would update for individual openQA jobs. This would also require a probably non-trivial extension of the dashboard database schema to store this information. I'm also not sure whether it is a good idea to query (potentially a lot of) openQA comments in this place.

Maybe it would be better if we extended the "approve" command instead. It would not just approve or skip the approval but also communicate the approval/skip with the dashboard. This would require a bit of refactoring to so we keep track of non-approvable incidents and update the dashboard accordingly (instead of just filtering out non-approvable incidents from the set with some debug logging). The dashboard already has an API to update incidents. So we just need to add another column to the dashboard database schema, e.g. review_comment which would store why the incident was approved or not approved. This could be displayed as per AC1.

I still have to reverse-engineer how the "boxes" for each incident on http://dashboard.qam.suse.de/blocked are computed. It seems one box represents one or more openQA jobs, though. So in order to show whether an @review:acceptable_for is relevant for a "box" as per AC2 just knowing why an incident was approved or not is not sufficient. For this we needed to track this information per openQA job as I tried with https://github.com/openSUSE/qem-bot/pull/190.

So to fulfill both ACs we might need both:

1. The bot tells the dashboard for each incident why it was approved or not (during the approval command) as some kind of review comment. This can be displayed on incident details pages (e.g. http://dashboard.qam.suse.de/incident/37356) like the approved and embargoed states.
2. The bot tells the dashboard for each and every openQA job whether @review:acceptable_for is present (during the sync or the approval command depending on what is more efficient). The dashboard stores this openQA-job-to-incident relation in its database and takes it into account when rendering "boxes" on http://dashboard.qam.suse.de/blocked.

I have already started with 2. but perhaps 1. is easier and a better starting point.

Maybe it would also make sense to cross-check with @jbaier_cz and @okurz whether this goes into the right direction.

jbaier_cz wrote in #note-13:

I hope I understood the initial motivation correctly. Take a look at some example, at this moment 37819:libarchive failed in "Kernel Maintenance" and "Security Maintenance" job groups because there are 2 failed openQA tests there (the number should be clear from the boxes, 1 test in each job group). The bot will of course not approve that incident, because there are failed jobs. Now, let's imagine that the job from the Security is not related and a reviewer puts the @review:acceptable_for:37819 comment there. In the next run, the bot will still not approve the job (because there is still 1 other failed job); but the reviewer will still see 2 failed tests in the dashboard and will need to investigate both of them to find out one of them is already commented.

So I guess we can look at this issue as some kind of a soft-fail for openQA job maybe? The idea from the ticket is to make the red box yellow-green if the failed test has the comment (now the tricky part might be: what if there are more failed tests in one box and only some of them are "acceptable", probably a yellow-red box?).

For me, the only requirement is that jobs with @review:acceptable_for:... will be counted as passed for the given incident. Highlighting label use with fancy colors is not important.

So do we have an example? As per AC2 I would expect to see this pretty clearly on http://dashboard.qam.suse.de/blocked but maybe there is no relevant case?

@MDoucha Would you have an example for us?

I called SQL select job_id,test,build,c.t_created,text from comments c join jobs j on j.id = c.job_id where text ~ 'acceptable_for' and build !~ ':' order by t_created desc limit 3; and found only

  job_id  |        test         |   build    |      t_created      |                        text

----------+---------------------+------------+---------------------+----------------------------------------
-------------
 17048481 | docker_tests        | 20250313-1 | 2025-03-14 12:07:43 | @review:acceptable_for:incident_37848:b
sc#1239303\r+
          |                     |            |                     |
 17048482 | docker-stable_tests | 20250313-1 | 2025-03-14 12:07:37 | @review:acceptable_for:incident_37848:b
sc#1239303\r+
          |                     |            |                     |
 17048481 | docker_tests        | 20250313-1 | 2025-03-14 12:07:18 | @review:acceptable_for:incident_37835:b
sc#1239303\r+
          |                     |            |                     |
(3 rows)

so no recent use of acceptable_for within aggregate tests. There are some uses in incident tests where it's not that useful but maybe can still be checked:

openqa=> select job_id,test,build,c.t_created,text from comments c join jobs j on j.id = c.job_id where text ~ 'acceptable_for' order by t_created desc limit 3;
  job_id  |            test            |       build       |      t_created      |
                         text
----------+----------------------------+-------------------+---------------------+--------------------------
---------------------------------------------------------------------------------
 17145817 | mau-extratests-virt-hyperv | :37973:dtb-armv7l | 2025-03-26 01:45:01 | @review:acceptable_for:in
cident_37973:automation issue, job pass in https://openqa.suse.de/tests/17106789
 17137273 | qam-klp_xfstests_btrfs     | :37973:dtb-armv7l | 2025-03-24 15:46:01 | @review:acceptable_for:in
cident_37973:known_issues
 17106981 | ltp_syscalls_m32           | :37973:dtb-armv7l | 2025-03-24 15:45:31 | @review:acceptable_for:in
cident_37973:bsc#1229648
(3 rows)

The bot-ng changes cannot be deployed/used right now.

So select * from job_remarks limit 1; (see https://gitlab.suse.de/qa-maintenance/bot-ng/-/blob/master/Readme.md#cleanup-of-unwanted-test-results) still shows no results despite over 300 results for select count(c.job_id) from comments as c join jobs j on j.id = c.job_id where text ~ 'acceptable_for'; and Ignoring failed job logged on approve jobs.

https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/4043079#L180 now ran with your latest qem-bot changes but no "job_remarks" in the db yet. What should we do next?

livdywan wrote in #note-19:

So do we have an example? As per AC2 I would expect to see this pretty clearly on http://dashboard.qam.suse.de/blocked but maybe there is no relevant case?

@MDoucha Would you have an example for us?

There are no pending kernel updates at the moment so we'll have to wait for the next update round in ~2 weeks.

I checked a few recent approve jobs but there are still none with "Ignoring failed job" and there are also no remarks in the database. I'd just wait until something comes up.

EDIT: I will use sapworker2-sp.qe.nue2.suse.org which is powered off anyway to create many worker instances.