action #57452
closedAutomatic summary of failures
0%
Description
I would like to have a tool, which would provide me quick and easy overview of fails of testcases.
It would be also able to check in git (git blame) who changed the failing line (yes, there are plenty of ways why this is not accurate, I'm open to better suggestions) to identify the person who could know the code best (definitely better than "Maintainer" put in the testcase).
Ideally, it would be also sending e-mails to the identified people.
They would then be expected to either fix or delegate fixing of the testcase.
Statistics collection would also be nice feature to help us to measure and predict future development by extrapolation.
I believe that this tool should be part of the openQA rather than another support tool.
Updated by coolo over 4 years ago
- Project changed from openQA Infrastructure to openQA Project
- Target version deleted (
Ready)
That implies that for failures always test authors are to be blamed. I would disagree
Updated by mkittler over 4 years ago
At least the "blamed" person touched the test code recently and therefore maybe understands what it does. So the test writer is likely be able to decide whether it is a test/product/infrastructure/... bug.
Updated by mloviska over 4 years ago
mkittler wrote:
At least the "blamed" person touched the test code recently and therefore maybe understands what it does. So the test writer is likely be able to decide whether it is a test/product/infrastructure/... bug.
Exactly! That should be another mechanism to ease the daily test reviewer's job.
Updated by okurz over 4 years ago
- Related to coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added
Updated by szarate over 4 years ago
mloviska wrote:
mkittler wrote:
At least the "blamed" person touched the test code recently and therefore maybe understands what it does. So the test writer is likely be able to decide whether it is a test/product/infrastructure/... bug.
Exactly! That should be another mechanism to ease the daily test reviewer's job.
not exactly what you're looking for, but there's something or there was an idea in the past... https://gitlab.suse.de/pgeorgiadis/ToelpelCI
Updated by livdywan over 4 years ago
szarate wrote:
mloviska wrote:
mkittler wrote:
At least the "blamed" person touched the test code recently and therefore maybe understands what it does. So the test writer is likely be able to decide whether it is a test/product/infrastructure/... bug.
I'd suggest we consider "persons involved" or "knowledgeable persons" as a more general term here. This would be one or more persons who recently touched the code, but also the specified maintainer of a test. In that sense it would be about making it easier to reach out to relevant people for help. You want someone who knows the code, not necessarily the person who caused the problem.
So I'd suggest "persons involved" would be a) maintainer b) most recent test developer(s), where ideally this would be distinguished accordingly.
not exactly what you're looking for, but there's something or there was an idea in the past... https://gitlab.suse.de/pgeorgiadis/ToelpelCI
That project seems to be about automatic verification, so I don't think it's what is being asked for here. Although I forwarded the link to another proposal that is about that topic. Thanks!
Updated by coolo over 4 years ago
- Status changed from New to Rejected
I spent quite some time lately reviewing test failures in maintenance and I can tell you: I'll reject this feature.
Why? to make this useful for you, you need to include tons of informations that are not known to openQA. E.g. LTP versions changes, changes to used test packages, changes to test suite settings, new incidents...
So to reach what you want to reach, you could just as well email everyone involved in git history between last good and today and include every incident manager - and still wouldn't reach any meaningful output in case of untracked assets used in the tests (as in www.suse.com, LTP, SCC, ...). As such your generated mails would end up in trash quicker than you wrote your tool.
Updated by okurz over 4 years ago
- Category set to Feature requests
I agree that the proposed tool is a bit too specific to make sense for the openQA project but the overall goal that you want to reach that I assume behind the idea should still be valid :) Also see #39719 for that
Updated by okurz over 4 years ago
- Related to coordination #58184: [saga][epic][use case] full version control awareness within openQA added