Project

General

Profile

action #57452

Automatic summary of failures

Added by vpelcak almost 2 years ago. Updated almost 2 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
-
Start date:
2019-09-27
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

I would like to have a tool, which would provide me quick and easy overview of fails of testcases.
It would be also able to check in git (git blame) who changed the failing line (yes, there are plenty of ways why this is not accurate, I'm open to better suggestions) to identify the person who could know the code best (definitely better than "Maintainer" put in the testcase).
Ideally, it would be also sending e-mails to the identified people.

They would then be expected to either fix or delegate fixing of the testcase.

Statistics collection would also be nice feature to help us to measure and predict future development by extrapolation.

I believe that this tool should be part of the openQA rather than another support tool.


Related issues

Related to openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issuesBlocked2017-01-232021-09-24

Related to openQA Project - coordination #58184: [saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches, fully versioned test schedules and configuration settingsBlocked2018-03-23

History

#1 Updated by coolo almost 2 years ago

  • Project changed from openQA Infrastructure to openQA Project
  • Target version deleted (Ready)

That implies that for failures always test authors are to be blamed. I would disagree

#2 Updated by mkittler almost 2 years ago

At least the "blamed" person touched the test code recently and therefore maybe understands what it does. So the test writer is likely be able to decide whether it is a test/product/infrastructure/... bug.

#3 Updated by mloviska almost 2 years ago

mkittler wrote:

At least the "blamed" person touched the test code recently and therefore maybe understands what it does. So the test writer is likely be able to decide whether it is a test/product/infrastructure/... bug.

Exactly! That should be another mechanism to ease the daily test reviewer's job.

#4 Updated by okurz almost 2 years ago

  • Related to coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added

#5 Updated by szarate almost 2 years ago

mloviska wrote:

mkittler wrote:

At least the "blamed" person touched the test code recently and therefore maybe understands what it does. So the test writer is likely be able to decide whether it is a test/product/infrastructure/... bug.

Exactly! That should be another mechanism to ease the daily test reviewer's job.

not exactly what you're looking for, but there's something or there was an idea in the past... https://gitlab.suse.de/pgeorgiadis/ToelpelCI

#6 Updated by cdywan almost 2 years ago

szarate wrote:

mloviska wrote:

mkittler wrote:

At least the "blamed" person touched the test code recently and therefore maybe understands what it does. So the test writer is likely be able to decide whether it is a test/product/infrastructure/... bug.

I'd suggest we consider "persons involved" or "knowledgeable persons" as a more general term here. This would be one or more persons who recently touched the code, but also the specified maintainer of a test. In that sense it would be about making it easier to reach out to relevant people for help. You want someone who knows the code, not necessarily the person who caused the problem.

So I'd suggest "persons involved" would be a) maintainer b) most recent test developer(s), where ideally this would be distinguished accordingly.

not exactly what you're looking for, but there's something or there was an idea in the past... https://gitlab.suse.de/pgeorgiadis/ToelpelCI

That project seems to be about automatic verification, so I don't think it's what is being asked for here. Although I forwarded the link to another proposal that is about that topic. Thanks!

#7 Updated by vpelcak almost 2 years ago

  • Description updated (diff)

#8 Updated by coolo almost 2 years ago

  • Status changed from New to Rejected

I spent quite some time lately reviewing test failures in maintenance and I can tell you: I'll reject this feature.

Why? to make this useful for you, you need to include tons of informations that are not known to openQA. E.g. LTP versions changes, changes to used test packages, changes to test suite settings, new incidents...

So to reach what you want to reach, you could just as well email everyone involved in git history between last good and today and include every incident manager - and still wouldn't reach any meaningful output in case of untracked assets used in the tests (as in www.suse.com, LTP, SCC, ...). As such your generated mails would end up in trash quicker than you wrote your tool.

#9 Updated by okurz almost 2 years ago

  • Category set to Feature requests

I agree that the proposed tool is a bit too specific to make sense for the openQA project but the overall goal that you want to reach that I assume behind the idea should still be valid :) Also see #39719 for that

#10 Updated by okurz almost 2 years ago

  • Related to coordination #58184: [saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches, fully versioned test schedules and configuration settings added

Also available in: Atom PDF