action #105408
closed
dashboard.qam.suse.de - add list for failures of test modules
Added by hurhaj almost 3 years ago.
Updated almost 2 years ago.
Category:
Feature requests
Description
User story¶
As a QEM's openQA reviewer, I want to easily find every test suite where a specific test module is failing, to have a better overview of the overall state of QEM's scenarios and identify possible regressions faster.
Acceptance criteria¶
- AC1: the list of failing test modules with affected test suites and related maint. updates (example below for illustration)
test module | failing in | blocking
-----------------------------------------------
-----------------------------------------------
systemd.pm | job_group_A, | S:M:1234:56789,
| job_group_B | S:M:2345:67890
-----------------------------------------------
bind.pm | job_group_G | S:M:3456:78901
Tasks¶
- figure out how to collect data
- create additional tab on dashboard
Further details¶
As per previous conversation with @okurz, this might be also useful for maintainers to check on their tests. The proposed second column could be visually split horizontally to QEM and product QE sections, so that they could see it all in one place.
- Category set to Feature requests
- Target version set to future
okurz wrote:
Your user story makes sense but I wonder why you wish for this to be present in dashboard.qam.suse.de? Links like for example https://openqa.suse.de/tests/overview?result=failed&modules=patch_and_reboot&modules_result=failed&distri=sle to show all jobs where a module "patch_and_reboot" fails in SLE tests for all versions, architectures, flavors already works.
Important part for me during a review is to have good overview. List like I described would show me the possible pattern of failures, while editing link would mean I have to start spotting it first and then confirming it by such links.
hurhaj wrote:
Important part for me during a review is to have good overview. List like I described would show me the possible pattern of failures, while editing link would mean I have to start spotting it first and then confirming it by such links.
I see. Well I am convinced that dashboard.qam.suse.de will never become a good tool to get statistics about openQA tests in the past. The focus is to show the present state and only that. All data within the dashboard is considered transient. How about https://maintenance-statistics.dyn.cloud.suse.de/dashboard/57-osd as a starting point showing "Which openQA test modules fail most often" as well as "which testsuites fail the most on OSD".
okurz wrote:
I see. Well I am convinced that dashboard.qam.suse.de will never become a good tool to get statistics about openQA tests in the past. The focus is to show the present state and only that. All data within the dashboard is considered transient. How about https://maintenance-statistics.dyn.cloud.suse.de/dashboard/57-osd as a starting point showing "Which openQA test modules fail most often" as well as "which testsuites fail the most on OSD".
But for the purposes of the review I am interested in present only. There is zero need for statistical data. With that, maybe...
** AC2: ** List shows only modules that have failed in the latest runs.
On the contrary, during review I am only interested in the tests that "are failing right now". If retry or next run passes, module should disappear from the list. To better illustrate, we currently have "Blocked" tab that groups by incident (maint. update) and shows related groups. What I want is to go one level deeper - to test modules in those groups, and list it by those.
And yes, I am aware that single incidents will be a problem again, as they don't really have next run. So they will need some special treatment again. My guess would be based on whether maint update is still active.
Hi @okurz. After revisiting this poo one year later, I have to admit I no longer see much use for this. Unless you believe it's still relevant from your point of view, I suggest rejecting.
- Status changed from New to Rejected
- Assignee set to okurz
Also available in: Atom
PDF