Project

General

Profile

Actions

action #105408

closed

dashboard.qam.suse.de - add list for failures of test modules

Added by hurhaj about 2 years ago. Updated about 1 year ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2022-01-25
Due date:
% Done:

0%

Estimated time:

Description

User story

As a QEM's openQA reviewer, I want to easily find every test suite where a specific test module is failing, to have a better overview of the overall state of QEM's scenarios and identify possible regressions faster.

Acceptance criteria

  • AC1: the list of failing test modules with affected test suites and related maint. updates (example below for illustration)
test module | failing in     | blocking
-----------------------------------------------
-----------------------------------------------
systemd.pm  | job_group_A,   | S:M:1234:56789,
            | job_group_B    | S:M:2345:67890
-----------------------------------------------
bind.pm     | job_group_G    | S:M:3456:78901

Tasks

  • figure out how to collect data
  • create additional tab on dashboard

Further details

As per previous conversation with @okurz, this might be also useful for maintainers to check on their tests. The proposed second column could be visually split horizontally to QEM and product QE sections, so that they could see it all in one place.

Actions #1

Updated by okurz about 2 years ago

  • Category set to Feature requests
  • Target version set to future

Your user story makes sense but I wonder why you wish for this to be present in dashboard.qam.suse.de? Links like for example https://openqa.suse.de/tests/overview?result=failed&modules=patch_and_reboot&modules_result=failed&distri=sle to show all jobs where a module "patch_and_reboot" fails in SLE tests for all versions, architectures, flavors already works.

Actions #2

Updated by hurhaj about 2 years ago

okurz wrote:

Your user story makes sense but I wonder why you wish for this to be present in dashboard.qam.suse.de? Links like for example https://openqa.suse.de/tests/overview?result=failed&modules=patch_and_reboot&modules_result=failed&distri=sle to show all jobs where a module "patch_and_reboot" fails in SLE tests for all versions, architectures, flavors already works.

Important part for me during a review is to have good overview. List like I described would show me the possible pattern of failures, while editing link would mean I have to start spotting it first and then confirming it by such links.

Actions #3

Updated by okurz about 2 years ago

hurhaj wrote:

Important part for me during a review is to have good overview. List like I described would show me the possible pattern of failures, while editing link would mean I have to start spotting it first and then confirming it by such links.

I see. Well I am convinced that dashboard.qam.suse.de will never become a good tool to get statistics about openQA tests in the past. The focus is to show the present state and only that. All data within the dashboard is considered transient. How about https://maintenance-statistics.dyn.cloud.suse.de/dashboard/57-osd as a starting point showing "Which openQA test modules fail most often" as well as "which testsuites fail the most on OSD".

Actions #4

Updated by hurhaj about 2 years ago

okurz wrote:

I see. Well I am convinced that dashboard.qam.suse.de will never become a good tool to get statistics about openQA tests in the past. The focus is to show the present state and only that. All data within the dashboard is considered transient. How about https://maintenance-statistics.dyn.cloud.suse.de/dashboard/57-osd as a starting point showing "Which openQA test modules fail most often" as well as "which testsuites fail the most on OSD".

But for the purposes of the review I am interested in present only. There is zero need for statistical data. With that, maybe...
** AC2: ** List shows only modules that have failed in the latest runs.

On the contrary, during review I am only interested in the tests that "are failing right now". If retry or next run passes, module should disappear from the list. To better illustrate, we currently have "Blocked" tab that groups by incident (maint. update) and shows related groups. What I want is to go one level deeper - to test modules in those groups, and list it by those.

And yes, I am aware that single incidents will be a problem again, as they don't really have next run. So they will need some special treatment again. My guess would be based on whether maint update is still active.

Actions #5

Updated by okurz about 2 years ago

We discussed in jitsi and found some ideas that we could do. https://openqa.suse.de/tests/overview?result=failed&distri=sle&build=20171108-1 is a good starting point but we would need to compute the build number. This could be done in qem-dashboard to have a clickable link. As alternative our maintenance aggregate tests are commonly defined by having the word "Updates" in the flavor name but we have different flavors. So would be nice to have a wildcard filter like https://openqa.suse.de/tests/overview?distri=sle&flavor=*Updates&result=failed which would resolve to all matching flavors, e.g. AZURE-BYOS-Updates and SAP-DVD-Updates and Server-DVD-Updates.

Additionally the count of module appearance would help. E.g. currently on https://openqa.suse.de/tests/overview?distri=sle&build=20220125-1&result=failed we see 7 times "register_system", 2 times "Sysctl". So maybe a list at the bottom listing the numbers in a list, by descending value.

Actions #6

Updated by hurhaj over 1 year ago

Hi @okurz. After revisiting this poo one year later, I have to admit I no longer see much use for this. Unless you believe it's still relevant from your point of view, I suggest rejecting.

Actions #7

Updated by okurz about 1 year ago

  • Status changed from New to Rejected
  • Assignee set to okurz

Ok, thanks for reviewing

Actions

Also available in: Atom PDF