https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842022-11-02T08:37:15ZopenSUSE Project Management ToolopenQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=5698602022-11-02T08:37:15Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed child" href="/issues/117655">action #117655</a>: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does size:M</i> added</li></ul> openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=5698662022-11-02T08:37:30Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Blocked</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p><a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655">#117655</a> first</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=5738612022-11-11T14:35:46Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Blocked</i> to <i>New</i></li><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li><li><strong>% Done</strong> changed from <i>30</i> to <i>0</i></li></ul> openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=5739572022-11-13T21:07:23Zokurzokurz@suse.com
<ul><li><strong>Target version</strong> changed from <i>Ready</i> to <i>future</i></li></ul> openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=5791062022-11-29T10:10:44Zmgrifalconi
<ul></ul><p>Hello, we were thinking on something similar but on the dashboard.qam.suse.de since it feels the point of integration between openQA and SMELT. But any place is good, important is to fullfill the goal :)</p>
<p>As a reviewer (for my squad or openQA review task) I would like to see:</p>
<ul>
<li>all failures related to a squad</li>
<li>ordered by how many release requests they are blocking (to help prioritize)</li>
<li>for how many runs (or days) they have been failing</li>
</ul>
<p>From that view, we can also start collecting long term metrics, maybe with a CI in gitlab and feed into Grafana<br>
Ideas of metrics:</p>
<ul>
<li>failures/day</li>
<li>days before a failure is fixed</li>
<li>number of release requests blocked</li>
<li>morning vs evening failures (how much manual work was done to improve situation)</li>
</ul>
<p>What do you think? How can we contribute?</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=5962062023-01-23T14:33:24Zszarate
<ul><li><strong>Parent task</strong> changed from <i>#114929</i> to <i>#118639</i></li></ul><p>Changing the parent task to reflect the reality better</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6459442023-06-22T12:59:13Zokurzokurz@suse.com
<ul><li><strong>Target version</strong> changed from <i>future</i> to <i>Ready</i></li></ul><p>I discussed this with szarate, PO of QE-Core, today and I feel it's the right time that we try to look into this.</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6516922023-07-12T09:50:36Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>[spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad"</i> to <i>[spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:S</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/651692/diff?detail_id=611507">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li></ul> openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6636232023-08-21T15:26:24Zkraihsebastian.riedel@suse.com
<ul><li><strong>Assignee</strong> set to <i>kraih</i></li></ul><p>Having a hard time deciding if this should be an openQA or dashboard feature, so some more clarification is needed. How exactly can we connect a review squad to an openQA job in the data? We have information like job group names and incidents in the metadata, but how do we match that to specific squads? Is that information present in the data at all yet?</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6636382023-08-21T16:18:51Zszarate
<ul></ul><p>Adding Michael as a watcher,</p>
<p>Right now, we don't have a mapping per see; there's an attempt done by Michael that tries <a href="https://gitlab.suse.de/qe-core/qem-dashboard-metric-miner/-/blob/main/main.go#L91" class="external">to tackle the issue</a>, however I think nothing keeps us from using the metadata project to map jobgroups to teams (which could backfire) or reuse the <a href="https://gitlab.suse.de/qa-maintenance/qam-openqa-yml/-/blob/master/qam_jobgroups.yml" class="external">qam_jobgroups</a> either by recompiling once the file changes or reading the file directly from the repository (we'd have to add a new property to the yaml), so no jobgroup will be left without an owner. </p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6636592023-08-21T18:48:31Zokurzokurz@suse.com
<ul></ul><p>kraih wrote in <a href="#note-10">#note-10</a>:</p>
<blockquote>
<p>Having a hard time deciding if this should be an openQA or dashboard feature, so some more clarification is needed.</p>
</blockquote>
<p>This is a clear openQA feature as it is described. But this is also why it's a timeboxed task. For example if we end up with a good understanding why it's not possible or why it does not make sense to have something like this in openQA then we can can still consider a feature for the qem-dashboard</p>
<blockquote>
<p>How exactly can we connect a review squad to an openQA job in the data?</p>
</blockquote>
<p>In a first approach just what the first ticket suggestion says: "filter jobs on /tests based on job group and job name", with a regex</p>
<blockquote>
<p>We have information like job group names and incidents in the metadata, but how do we match that to specific squads? Is that information present in the data at all yet?</p>
</blockquote>
<p>Don't consider the SUSE QAM metadata for this. That would be something different.</p>
<p>What szarate said could be something like "filter by regex match on job group description". Then anybody could put any custom field there and match on it</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6638482023-08-22T10:34:07Zkraihsebastian.riedel@suse.com
<ul></ul><p>okurz wrote in <a href="#note-12">#note-12</a>:</p>
<blockquote>
<p>This is a clear openQA feature as it is described. But this is also why it's a timeboxed task. For example if we end up with a good understanding why it's not possible or why it does not make sense to have something like this in openQA then we can can still consider a feature for the qem-dashboard</p>
</blockquote>
<p>Yes, i'm trying to understand what the users actually need here, and this is starting to look like two separate features (for followup tickets). For the openQA side we don't really want to introduce a "squad" concept, which is very SUSE org specific, just to be able to search for unreviewed jobs. This could theoretically already be done with the existing data, as you said, with the TODO option and some regex matching of job group names. That's what i'll prototype for this timeboxed ticket.</p>
<p>The second feature is potentially for the dashboard, since it is about displaying the status of active incidents. The users would also like to have an overview of which squads are currently blocking which incidents. Squads can once again be identified from job group names here, but we do not yet store enough openQA job metadata to know which jobs have been reviewed. The qem-bot retrieves openQA job comments and parses them to get that information, but does not forward it to the dashboard yet. Theoretically that would be all we need to make per squad blocked incident pages for the dashboard.</p>
<p>The squad/job group name mapping data from existing tooling:</p>
<pre><code>var squadNames = []string{"SAP/HA", "Kernel", "QAC", "Yast", "Security", "Core"}
var squads = map[string][]string{
"SAP/HA": {"SAP", " HA"},
"Kernel": {"Kernel", " HPC"},
"QAC": {"Public", "Containers", "JeOS", "Micro", "Wicked", "SLEM"},
"Yast": {"YaST"},
"Security": {"Security"},
"Core": {"Core", "TERADATA", "SLE 15", "SLE 12"},
}
</code></pre> openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6638512023-08-22T10:53:16Zmgrifalconi
<ul></ul><blockquote>
<p>we do not yet store enough openQA job metadata to know which jobs have been reviewed.</p>
</blockquote>
<p>Please mind that we (openQA reviewers) do not care if there is a comment, a poo# or a bsc# attached to a test failure. That is not good enough and still needs a high priority action to handle the failure.</p>
<p>As long as there is a failure the linked RR will be blocked, so we ask squads to take a decision between:</p>
<ul>
<li>fixing the test issue and getting the job green</li>
<li>softfailing the test while investigating/fixing and knowing that the test is not broken due to a regression</li>
<li>asking to reject the RR </li>
</ul>
<p>So I would not want to hide "reviewed" failures to squads, since they bother us (and maintenance coordination) as much as a "unreviewed" failure</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6638632023-08-22T11:28:25Zkraihsebastian.riedel@suse.com
<ul></ul><p>mgrifalconi wrote in <a href="#note-14">#note-14</a>:</p>
<blockquote>
<p>Please mind that we (openQA reviewers) do not care if there is a comment, a poo# or a bsc# attached to a test failure. That is not good enough and still needs a high priority action to handle the failure.</p>
<p>As long as there is a failure the linked RR will be blocked, so we ask squads to take a decision between:</p>
<ul>
<li>fixing the test issue and getting the job green</li>
<li>softfailing the test while investigating/fixing and knowing that the test is not broken due to a regression</li>
<li>asking to reject the RR </li>
</ul>
</blockquote>
<p>That is good to know. Then theoretically we should already have enough data and it's just a matter of deciding how to present it on the dashboard with squad specific pages. Good start would probably be to introduce the "squad" concept on the "Blocked" page and to allow filtering by squad. And perhaps to expose squad data in the dashboard API so the mapping data doesn't have to maintained in multiple places.</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6638752023-08-22T12:17:38Zszarate
<ul></ul><p>kraih wrote in <a href="#note-15">#note-15</a>:</p>
<blockquote>
<p>mgrifalconi wrote in <a href="#note-14">#note-14</a>:</p>
<blockquote>
<p>Please mind that we (openQA reviewers) do not care if there is a comment, a poo# or a bsc# attached to a test failure. That is not good enough and still needs a high priority action to handle the failure.</p>
<p>As long as there is a failure the linked RR will be blocked, so we ask squads to take a decision between:</p>
<ul>
<li>fixing the test issue and getting the job green</li>
<li>softfailing the test while investigating/fixing and knowing that the test is not broken due to a regression</li>
<li>asking to reject the RR </li>
</ul>
</blockquote>
<p>That is good to know. Then theoretically we should already have enough data and it's just a matter of deciding how to present it on the dashboard with squad specific pages. Good start would probably be to introduce the "squad" concept on the "Blocked" page and to allow filtering by squad. And perhaps to expose squad data in the dashboard API so the mapping data doesn't have to maintained in multiple places.</p>
</blockquote>
<p>I think that the Squad concept can be mapped to review groups, teams, review teams or user groups... after all, squads is more of an arbitrary concept that comes from <a href="https://www.atlassian.com/agile/agile-at-scale/spotify" class="external">Agile at Scale</a> than for how systems are built.</p>
<p>Here I mean that job groups could be assigned to a group, and a user could belong to a group, however bringing the stuff into the dashboard itself has its own advantages... like making #127208 obsolete :)</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6639652023-08-22T15:01:05Zkraihsebastian.riedel@suse.com
<ul></ul><blockquote>
<p>Here I mean that job groups could be assigned to a group, and a user could belong to a group, however bringing the stuff into the dashboard itself has its own advantages... like making #127208 obsolete :)</p>
</blockquote>
<p>That works too, and would be fairly trivial to implement if we reuse id.opensuse.org for identity management again. Query parameters for existing filters would probably still be a good idea to have though.</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6647392023-08-24T14:34:29Zkraihsebastian.riedel@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li></ul><p>Making a proof of concept for filtering by job group names.</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6648262023-08-25T04:09:18Zopenqa_reviewopenqa-review@suse.de
<ul><li><strong>Due date</strong> set to <i>2023-09-08</i></li></ul><p>Setting due date based on mean cycle time of SUSE QE Tools</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6659932023-08-29T15:10:47Zkraihsebastian.riedel@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>Draft PR with possible spike solution: <a href="https://github.com/os-autoinst/openQA/pull/5291" class="external">https://github.com/os-autoinst/openQA/pull/5291</a></p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6670192023-08-31T11:45:09Zkraihsebastian.riedel@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed child" href="/issues/134933">action #134933</a>: Filter openQA todo-jobs on /tests belonging to one "review squad" size:M</i> added</li></ul> openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6671542023-08-31T14:18:50Zkraihsebastian.riedel@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul> openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6671572023-08-31T14:19:28Zkraihsebastian.riedel@suse.com
<ul></ul><p>Followup ticket: <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Filter openQA todo-jobs on /tests belonging to one "review squad" size:M (Resolved)" href="https://progress.opensuse.org/issues/134933">#134933</a></p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6675022023-09-01T10:00:52Zkraihsebastian.riedel@suse.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>In Progress</i></li></ul><p>Back into progress since the PR was for <code>/tests/overview</code> while the feature was supposed to be limited to <code>/tests</code>.</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=6676702023-09-01T15:20:08Zkraihsebastian.riedel@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>The followup ticket will be about both endpoints.</p>
openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=7522872024-01-16T13:58:23Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/153646">action #153646</a>: [qe-core] Rename incident job groups owned by qe-core</i> added</li></ul> openQA Project - action #119746: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:Shttps://progress.opensuse.org/issues/119746?journal_id=7536372024-01-18T12:01:44Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> deleted (<del><i>2023-09-08</i></del>)</li></ul>