QA » openQA Project

This would make the "All tests" page much less "All tests" like, e.g. I assume tables for running and scheduled jobs would be hidden completely.

Due to performance problems the number of finished jobs is by default limited to 500. Most existing filters on that page are implemented client-side and hence only limit these 500 jobs further. How is the new filter supposed to work? Also only within the most recent 500 jobs or should it go deeper so we would really show up to 500 unreviewed jobs? Then it needed to be implemented server-side like the existing "Show only relevant jobs" filter (but still needed to handle query parameters like the filter for result types). I'm wondering how fast it would be, though. Going down within the job history (while parsing job comments for ticket references using Perl regex) until 500 unreviewed jobs have been found might take a while.

mkittler wrote:

Due to performance problems the number of finished jobs is by default limited to 500. Most existing filters on that page are implemented client-side and hence only limit these 500 jobs further. How is the new filter supposed to work? Also only within the most recent 500 jobs or should it go deeper so we would really show up to 500 unreviewed jobs? Then it needed to be implemented server-side like the existing "Show only relevant jobs" filter (but still needed to handle query parameters like the filter for result types).

I read the ticket to mean the API will filter the results based on the query - or is this actually all done client-side now? Your comment sounds like we have a bit of both.

Your comment sounds like we have a bit of both.

That's also the case. Filtering by result type and the search box are applied on the client-side (and only reduce the 500 jobs returned by the server).

By the way, I'm wondering when this would be useful. Maybe it is just me but I'm only using the "All tests" page on the production instances for running and scheduled jobs (and I suppose these tables would be hidden completely or shown unaffected). The list of finished jobs is rather arbitrary and due to the limit it goes only down to a few hours in the past. (Of course on my personal openQA instance it is another story - but I double people do much review work on their personal instances.)

Sure it can be slower than the default of just the latest 500 jobs but definitely not slower than users needing to manually open /tests/overview of 10 products :)

But it wouldn't be useful if the request times out, e.g. https://openqa.opensuse.org/tests/overview shows lots of tests on o3 but adding the TODO parameter here (https://openqa.opensuse.org/tests/overview?todo=1) results in a gateway timeout. I'm just saying that searching for 500 unreviewed jobs might take long, e.g. in the extreme case when all jobs have actually already been reviewed the whole list of jobs within the database needed to be checked (in order to return zero results).

If you are not sure what we want to achieve or why we want to do this please see the "Motivation" …

The motivation "I want to find unreviewed openQA test failures on /tests to have access to a single page showing all failures needing review work" really raises the question: Unreviewed openQA tests since when? I'm sure users don't want to bother with very old failed jobs which have no relevance anymore.

mkittler wrote:

Sure it can be slower than the default of just the latest 500 jobs but definitely not slower than users needing to manually open /tests/overview of 10 products :)

But it wouldn't be useful if the request times out, e.g. https://openqa.opensuse.org/tests/overview shows lots of tests on o3 but adding the TODO parameter here (https://openqa.opensuse.org/tests/overview?todo=1) results in a gateway timeout. I'm just saying that searching for 500 unreviewed jobs might take long, e.g. in the extreme case when all jobs have actually already been reviewed the whole list of jobs within the database needed to be checked (in order to return zero results).

yes, you are right. The difference to "manually open /tests/overview of 10 products" is that first we look up the latest job within each scenario and only then check if they are reviewed. So this is what we should do here as well. This is what I meant in the suggestions stating "Add […] todo=1 same as on /tests/overview . At best use the same logic to prevent duplication"

If you are not sure what we want to achieve or why we want to do this please see the "Motivation" …

The motivation "I want to find unreviewed openQA test failures on /tests to have access to a single page showing all failures needing review work" really raises the question: Unreviewed openQA tests since when? I'm sure users don't want to bother with very old failed jobs which have no relevance anymore.

Correct. As stated above we should look for the latest in each test scenario. And of course this can then reveal that "only jobs in the unstable development job group are left as unreviewed". This is why I suggested the optional "job group include/exclude regex" which is an optional implementation so depending on feasibility the implementors can decide if this is out of scope for the current story or not.

okurz wrote:

yes, you are right. The difference to "manually open /tests/overview of 10 products" is that first we look up the latest job within each scenario and only then check if they are reviewed. So this is what we should do here as well. This is what I meant in the suggestions stating "Add […] todo=1 same as on /tests/overview . At best use the same logic to prevent duplication"

The difference to /tests/overview is that the list of jobs to load is determined by the query parameters.

What would "latest job within each scenario" here mean for the /tests page?

What would "latest job within each scenario" here mean for the /tests page?

Same as already when you open /tests, the latest within each scenario is resolved. This means that older jobs for the same or older builds are hidden. For example if someone retriggers tests then only the latest clone is shown. Actually I am not so sure about "older builds" but I don't ask for any changes in this regard.

okurz wrote:

What would "latest job within each scenario" here mean for the /tests page?

Same as already when you open /tests, the latest within each scenario is resolved. This means that older jobs for the same or older builds are hidden. For example if someone retriggers tests then only the latest clone is shown. Actually I am not so sure about "older builds" but I don't ask for any changes in this regard.

Jobs for older builds are shown, as long as they are in the latest 500.

I don't know about others, but I still wouldn't know how to approach this, given that there is no trivial way to filter the "reviewed" property with SQL.
Getting the latest 500 failed via SQL and then filtering label/bugref with perl would work, and already this would require to fetch more data then for normal /tests queries (currently only bug counts are fetched).

tinita wrote:

What would "latest job within each scenario" here mean for the /tests page?

Let me rephrase: What do you mean with "scenario" on the /tests page? Running, scheduled and finished?

I'm still wondering how the scenario would come into play here. Determining all present scenarios (unique combinations of <distri>-<version>-<flavor>-<arch>-<test_suite>@<machine>) to determine their latest job seems pretty expensive to do (across all jobs).

Or do you expect to supply parameters for a specific scenario when the TODO parameter is used? (But then the all tests page would just be a worse structured version of the test result overview page.)

The description is up-to-date with what is currently expected so all previous comments can be ignored :)

waiting for #105001 . After that we can also present and discuss the TODO-functionality in a SUSE QE Tools workshop. Added a topic proposal on https://progress.opensuse.org/projects/qa/wiki/Wiki#Workshop-Topics