Project

General

Profile

Actions

coordination #93246

closed

QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

coordination #89062: [epic] Simplify review for SUSE QAM

[epic] List all unreviewed failed (or incomplete) jobs on /tests on request size:M

Added by okurz almost 3 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2021-12-09
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Motivation

As a test reviewer and member of one of many QA squads I want to find unreviewed openQA test failures on /tests to have access to a single page showing all failures needing review work.

Also see parent #89062

Acceptance criteria

  • AC1: A query parameter "todo=1" on /tests can be specified to show only jobs needing review, same as we do on /tests/overview

Suggestions

/tests can already be configured like https://openqa.opensuse.org/tests/?resultfilter=Failed&resultfilter=Incomplete&resultfilter=timeout_exceeded , even over UI elements.

  • Try t/ui/01-list.t and see if you can extend it
  • Add query parameter to filter only jobs that need review, e.g. "todo=1", same as on /tests/overview . At best use the same logic to prevent duplication

Out of scope

  • Don't care about the "running" or "scheduled" jobs on /tests as long as they at least show something useful when no manual query parameters are used or no filtering box selection has been made.
  • Performance improvements on /tests, as the feature should only be activated on explicit request by users
  • Don't care about the number of jobs that are displayed after all the filters apply. https://openqa.opensuse.org/tests?match=toolkits&resultfilter=Failed is one example showing that there can be enough "review candidates" depending on the specified test suite
  • Don't care about changing the limit of jobs (e.g. more than 500)
  • Don't care about filtering the job for other parameters like job group, scenario name, result, state, etc.

Subtasks 3 (0 open3 closed)

action #103765: Support for "todo" query parameter on /tests, same as /tests/overview size:MResolvedtinita2021-12-09

Actions
action #104995: Add UI element and help text for "todo" query parameter on /tests, similar as /tests/overviewResolvedmkittler

Actions
action #105001: Add doc for "todo" query parameter on /tests, similar to /tests/overview size:SResolvedkodymo2022-01-18

Actions

Related issues 1 (0 open1 closed)

Copied to openQA Project - action #94937: Distinguish comment types on jobs on /tests (maybe optional) size:SResolvedtinita2021-05-28

Actions
Actions #1

Updated by VANASTASIADIS almost 3 years ago

  • Target version set to Ready
Actions #2

Updated by mkittler almost 3 years ago

This would make the "All tests" page much less "All tests" like, e.g. I assume tables for running and scheduled jobs would be hidden completely.

Due to performance problems the number of finished jobs is by default limited to 500. Most existing filters on that page are implemented client-side and hence only limit these 500 jobs further. How is the new filter supposed to work? Also only within the most recent 500 jobs or should it go deeper so we would really show up to 500 unreviewed jobs? Then it needed to be implemented server-side like the existing "Show only relevant jobs" filter (but still needed to handle query parameters like the filter for result types). I'm wondering how fast it would be, though. Going down within the job history (while parsing job comments for ticket references using Perl regex) until 500 unreviewed jobs have been found might take a while.

Actions #3

Updated by livdywan almost 3 years ago

mkittler wrote:

Due to performance problems the number of finished jobs is by default limited to 500. Most existing filters on that page are implemented client-side and hence only limit these 500 jobs further. How is the new filter supposed to work? Also only within the most recent 500 jobs or should it go deeper so we would really show up to 500 unreviewed jobs? Then it needed to be implemented server-side like the existing "Show only relevant jobs" filter (but still needed to handle query parameters like the filter for result types).

I read the ticket to mean the API will filter the results based on the query - or is this actually all done client-side now? Your comment sounds like we have a bit of both.

Actions #4

Updated by mkittler almost 3 years ago

Your comment sounds like we have a bit of both.

That's also the case. Filtering by result type and the search box are applied on the client-side (and only reduce the 500 jobs returned by the server).


By the way, I'm wondering when this would be useful. Maybe it is just me but I'm only using the "All tests" page on the production instances for running and scheduled jobs (and I suppose these tables would be hidden completely or shown unaffected). The list of finished jobs is rather arbitrary and due to the limit it goes only down to a few hours in the past. (Of course on my personal openQA instance it is another story - but I double people do much review work on their personal instances.)

Actions #5

Updated by okurz almost 3 years ago

  • Description updated (diff)

mkittler wrote:

This would make the "All tests" page much less "All tests" like, e.g. I assume tables for running and scheduled jobs would be hidden completely.

For the user story as define I suggest to completely ignore the impact on "running/scheduled" tests. At this point if expert users deliberately use a query parameter, e.g. over a manual entry in the address bar or even over a checkbox, they know what they ask for. If it makes it easier for us then, yes, for this mode we could make the "running / scheduled" boxes completely invisible.

Due to performance problems the number of finished jobs is by default limited to 500. Most existing filters on that page are implemented client-side and hence only limit these 500 jobs further. How is the new filter supposed to work? Also only within the most recent 500 jobs or should it go deeper so we would really show up to 500 unreviewed jobs? Then it needed to be implemented server-side like the existing "Show only relevant jobs" filter (but still needed to handle query parameters like the filter for result types).

Yes, the initial server-side query needs to know this request.

I'm wondering how fast it would be, though. Going down within the job history (while parsing job comments for ticket references using Perl regex) until 500 unreviewed jobs have been found might take a while.

Sure it can be slower than the default of just the latest 500 jobs but definitely not slower than users needing to manually open /tests/overview of 10 products :)

By the way, I'm wondering when this would be useful. Maybe it is just me but I'm only using the "All tests" page on the production instances for running and scheduled jobs […] The list of finished jobs is rather arbitrary and due to the limit it goes only down to a few hours in the past.

Correct. Because I also think that power users on big openQA instances like openqa.opensuse.org and openqa.suse.de use /tests/overview I want to make /tests more useful for new use cases which we want to cover. If you are not sure what we want to achieve or why we want to do this please see the "Motivation" in the current ticket as well as the parent epic and saga with related tickets. I can also provide more context over other communication channels if you prefer.

(Of course on my personal openQA instance it is another story - but I double people do much review work on their personal instances.)

Actually I am sure that users rely more on /tests on smaller instances where "the last 500" make more sense, e.g. are only triggered by single users. So we must not break this use case.

Actions #6

Updated by mkittler almost 3 years ago

Sure it can be slower than the default of just the latest 500 jobs but definitely not slower than users needing to manually open /tests/overview of 10 products :)

But it wouldn't be useful if the request times out, e.g. https://openqa.opensuse.org/tests/overview shows lots of tests on o3 but adding the TODO parameter here (https://openqa.opensuse.org/tests/overview?todo=1) results in a gateway timeout. I'm just saying that searching for 500 unreviewed jobs might take long, e.g. in the extreme case when all jobs have actually already been reviewed the whole list of jobs within the database needed to be checked (in order to return zero results).

If you are not sure what we want to achieve or why we want to do this please see the "Motivation" …

The motivation "I want to find unreviewed openQA test failures on /tests to have access to a single page showing all failures needing review work" really raises the question: Unreviewed openQA tests since when? I'm sure users don't want to bother with very old failed jobs which have no relevance anymore.

Actions #7

Updated by okurz almost 3 years ago

mkittler wrote:

Sure it can be slower than the default of just the latest 500 jobs but definitely not slower than users needing to manually open /tests/overview of 10 products :)

But it wouldn't be useful if the request times out, e.g. https://openqa.opensuse.org/tests/overview shows lots of tests on o3 but adding the TODO parameter here (https://openqa.opensuse.org/tests/overview?todo=1) results in a gateway timeout. I'm just saying that searching for 500 unreviewed jobs might take long, e.g. in the extreme case when all jobs have actually already been reviewed the whole list of jobs within the database needed to be checked (in order to return zero results).

yes, you are right. The difference to "manually open /tests/overview of 10 products" is that first we look up the latest job within each scenario and only then check if they are reviewed. So this is what we should do here as well. This is what I meant in the suggestions stating "Add […] todo=1 same as on /tests/overview . At best use the same logic to prevent duplication"

If you are not sure what we want to achieve or why we want to do this please see the "Motivation" …

The motivation "I want to find unreviewed openQA test failures on /tests to have access to a single page showing all failures needing review work" really raises the question: Unreviewed openQA tests since when? I'm sure users don't want to bother with very old failed jobs which have no relevance anymore.

Correct. As stated above we should look for the latest in each test scenario. And of course this can then reveal that "only jobs in the unstable development job group are left as unreviewed". This is why I suggested the optional "job group include/exclude regex" which is an optional implementation so depending on feasibility the implementors can decide if this is out of scope for the current story or not.

Actions #8

Updated by okurz almost 3 years ago

  • Priority changed from High to Low

As confirmed by vpelcak in DM where he addressed me in https://chat.suse.de/direct/2FFxjQXvCCPbj5kbFYvRdSgu8mNq2StdnK?msg=qKsBM6NKNqghTkuEF we can focus on "openqa-review" related tasks with the expectation that this can act as the "better overview over their failing testcases" so we reduce priority here

Actions #9

Updated by tinita over 2 years ago

okurz wrote:

yes, you are right. The difference to "manually open /tests/overview of 10 products" is that first we look up the latest job within each scenario and only then check if they are reviewed. So this is what we should do here as well. This is what I meant in the suggestions stating "Add […] todo=1 same as on /tests/overview . At best use the same logic to prevent duplication"

The difference to /tests/overview is that the list of jobs to load is determined by the query parameters.

What would "latest job within each scenario" here mean for the /tests page?

Actions #10

Updated by okurz over 2 years ago

What would "latest job within each scenario" here mean for the /tests page?

Same as already when you open /tests, the latest within each scenario is resolved. This means that older jobs for the same or older builds are hidden. For example if someone retriggers tests then only the latest clone is shown. Actually I am not so sure about "older builds" but I don't ask for any changes in this regard.

Actions #11

Updated by tinita over 2 years ago

okurz wrote:

What would "latest job within each scenario" here mean for the /tests page?

Same as already when you open /tests, the latest within each scenario is resolved. This means that older jobs for the same or older builds are hidden. For example if someone retriggers tests then only the latest clone is shown. Actually I am not so sure about "older builds" but I don't ask for any changes in this regard.

Jobs for older builds are shown, as long as they are in the latest 500.

I don't know about others, but I still wouldn't know how to approach this, given that there is no trivial way to filter the "reviewed" property with SQL.
Getting the latest 500 failed via SQL and then filtering label/bugref with perl would work, and already this would require to fetch more data then for normal /tests queries (currently only bug counts are fetched).

Actions #12

Updated by tinita over 2 years ago

tinita wrote:

What would "latest job within each scenario" here mean for the /tests page?

Let me rephrase: What do you mean with "scenario" on the /tests page? Running, scheduled and finished?

Actions #13

Updated by okurz over 2 years ago

  • Subject changed from List all unreviewed failed (or incomplete) jobs on /tests to List all unreviewed failed (or incomplete) jobs on /tests on request
  • Description updated (diff)

tinita wrote:

okurz wrote:

What would "latest job within each scenario" here mean for the /tests page?

Same as already when you open /tests, the latest within each scenario is resolved. This means that older jobs for the same or older builds are hidden. For example if someone retriggers tests then only the latest clone is shown. Actually I am not so sure about "older builds" but I don't ask for any changes in this regard.

Jobs for older builds are shown, as long as they are in the latest 500.

I don't know about others, but I still wouldn't know how to approach this, given that there is no trivial way to filter the "reviewed" property with SQL.

Do you think the suggestions I made in the description regarding the "todo"-flag which we already have on /tests/overview does not help here?

Getting the latest 500 failed via SQL and then filtering label/bugref with perl would work, and already this would require to fetch more data then for normal /tests queries (currently only bug counts are fetched).

Well, I don't see a problem with that as I suggested that a new initial query is done based on query parameters. So this is not about "filtering pre-loaded data client-side with javascript" at this point.

tinita wrote:

tinita wrote:

What would "latest job within each scenario" here mean for the /tests page?

Let me rephrase: What do you mean with "scenario" on the /tests page? Running, scheduled and finished?

I mean scenario as described on
http://open.qa/docs/#concepts

<distri>-<version>-<flavor>-<arch>-<test_suite>@<machine>, e.g. "openSUSE-Tumbleweed-DVD-x86_64-gnome@64bit".

Actions #14

Updated by okurz over 2 years ago

  • Copied to action #94937: Distinguish comment types on jobs on /tests (maybe optional) size:S added
Actions #15

Updated by okurz over 2 years ago

  • Subject changed from List all unreviewed failed (or incomplete) jobs on /tests on request to List all unreviewed failed (or incomplete) jobs on /tests on request size:M
  • Description updated (diff)

estimated together

Actions #16

Updated by okurz over 2 years ago

  • Target version changed from Ready to future
Actions #17

Updated by okurz over 2 years ago

  • Subject changed from List all unreviewed failed (or incomplete) jobs on /tests on request size:M to List all unreviewed failed (or incomplete) jobs on /tests on request
  • Description updated (diff)
  • Status changed from Workable to New

#94937 solved. We can now continue here. To be re-estimated after prerequisities changed. Updated description.

Actions #18

Updated by mkittler over 2 years ago

I'm still wondering how the scenario would come into play here. Determining all present scenarios (unique combinations of <distri>-<version>-<flavor>-<arch>-<test_suite>@<machine>) to determine their latest job seems pretty expensive to do (across all jobs).

Or do you expect to supply parameters for a specific scenario when the TODO parameter is used? (But then the all tests page would just be a worse structured version of the test result overview page.)

Actions #19

Updated by okurz over 2 years ago

  • Target version changed from future to Ready
Actions #20

Updated by okurz over 2 years ago

  • Description updated (diff)
Actions #21

Updated by tinita over 2 years ago

  • Subject changed from List all unreviewed failed (or incomplete) jobs on /tests on request to List all unreviewed failed (or incomplete) jobs on /tests on request size:M
  • Status changed from New to Workable
Actions #22

Updated by okurz over 2 years ago

The description is up-to-date with what is currently expected so all previous comments can be ignored :)

Actions #23

Updated by okurz over 2 years ago

  • Tracker changed from action to coordination
  • Subject changed from List all unreviewed failed (or incomplete) jobs on /tests on request size:M to [epic] List all unreviewed failed (or incomplete) jobs on /tests on request size:M
  • Status changed from Workable to In Progress
  • Assignee set to okurz

after discussing this ticket in weekly estimation 2021-12-09 I understood that it might still not be clear what to do and that we should clarify some points, e.g. what "all" test mean and the relation to todo=1 and if that really offers a new use case for users.

Actions #24

Updated by okurz over 2 years ago

  • Status changed from In Progress to Blocked
Actions #25

Updated by okurz about 2 years ago

waiting for #105001 . After that we can also present and discuss the TODO-functionality in a SUSE QE Tools workshop. Added a topic proposal on https://progress.opensuse.org/projects/qa/wiki/Wiki#Workshop-Topics

Actions #26

Updated by okurz about 2 years ago

  • Status changed from Blocked to Resolved
Actions

Also available in: Atom PDF