Actions
coordination #41057
opencoordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA
[epic] Make reviewing results easier
Start date:
2018-04-16
Due date:
% Done:
33%
Estimated time:
(Total: 0.00 h)
Tags:
Description
Observation¶
openQA test in scenario opensuse-15.1-DVD-x86_64-update_Leap_15.0_gnome@64bit fails in
gnucash
So I was just staring at this gnucash failure without any hint what went wrong. So what I had to do is
- to go to the previous results
- observe that the last run was green
- click on the last run and manually navigate to gnucash
- have two browser windows open to visually compare the difference
- see that gnucash seems to behave the same but still fails. Maybe some subtle font change somewhere?
- look at the source code trying to understand what the test does
- can't understand why the code didn't look for a needle. wondering if the code I am looking at is what actually ran
- open a terminal to look at the git history of gnucash.pm
- see that okurz reverted a previous change earlier today
- restart all tests that use gnucash
I think there are opportunities to reduce the amount of manual steps involved, for example
- tell me that the previous run was good and provide a link I can click on to compare, jump to the relevant test/needle right away
- warn me that the code I am looking at is not what was run. provide link to show the diff
Bonus points
- show links/diffs to other runs in the same build that also have gnucash failures to proof it's always the same
- links/diffs to gnucash failures in other job groups. I suppose in this case the problem was spotted in TW already
- single click solution to restart all affected tests at once
Actions