coordination #41057
coordination #39719: [saga][epic] Detect "known failures" and mark jobs as such to make tests more stable, reviewing test results and tracking known issues easier
[epic] Make reviewing results easier
Start date:
2018-09-14
Due date:
% Done:
0%
Estimated time:
Difficulty:
hard
Description
Observation¶
openQA test in scenario opensuse-15.1-DVD-x86_64-update_Leap_15.0_gnome@64bit fails in
gnucash
So I was just staring at this gnucash failure without any hint what went wrong. So what I had to do is
- to go to the previous results
- observe that the last run was green
- click on the last run and manually navigate to gnucash
- have two browser windows open to visually compare the difference
- see that gnucash seems to behave the same but still fails. Maybe some subtle font change somewhere?
- look at the source code trying to understand what the test does
- can't understand why the code didn't look for a needle. wondering if the code I am looking at is what actually ran
- open a terminal to look at the git history of gnucash.pm
- see that okurz reverted a previous change earlier today
- restart all tests that use gnucash
I think there are opportunities to reduce the amount of manual steps involved, for example
- tell me that the previous run was good and provide a link I can click on to compare, jump to the relevant test/needle right away
- warn me that the code I am looking at is not what was run. provide link to show the diff
Bonus points
- show links/diffs to other runs in the same build that also have gnucash failures to proof it's always the same
- links/diffs to gnucash failures in other job groups. I suppose in this case the problem was spotted in TW already
- single click solution to restart all affected tests at once
Related issues
History
#1
Updated by lnussel over 2 years ago
- Project changed from openQA Tests to openQA Project
- Description updated (diff)
- Category changed from Bugs in existing tests to 124
#2
Updated by coolo over 2 years ago
- Subject changed from test fails in gnucash to Make reviewing results easier
#3
Updated by coolo over 2 years ago
- Subject changed from Make reviewing results easier to [EPIC] Make reviewing results easier
- Difficulty set to hard
#4
Updated by okurz over 1 year ago
- Category changed from 124 to Feature requests
#5
Updated by okurz about 1 year ago
- Related to coordination #19720: [epic] Simplify investigation of job failures added
#9
Updated by szarate 3 months ago
See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html