Project

General

Profile

coordination #41057

coordination #39719: [saga][epic] Detect "known failures" and mark jobs as such to make tests more stable, reviewing test results and tracking known issues easier

[epic] Make reviewing results easier

Added by lnussel over 2 years ago. Updated 3 months ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2018-09-14
Due date:
% Done:

0%

Estimated time:
Difficulty:
hard

Description

Observation

openQA test in scenario opensuse-15.1-DVD-x86_64-update_Leap_15.0_gnome@64bit fails in
gnucash

So I was just staring at this gnucash failure without any hint what went wrong. So what I had to do is

  • to go to the previous results
  • observe that the last run was green
  • click on the last run and manually navigate to gnucash
  • have two browser windows open to visually compare the difference
  • see that gnucash seems to behave the same but still fails. Maybe some subtle font change somewhere?
  • look at the source code trying to understand what the test does
  • can't understand why the code didn't look for a needle. wondering if the code I am looking at is what actually ran
  • open a terminal to look at the git history of gnucash.pm
  • see that okurz reverted a previous change earlier today
  • restart all tests that use gnucash

I think there are opportunities to reduce the amount of manual steps involved, for example

  • tell me that the previous run was good and provide a link I can click on to compare, jump to the relevant test/needle right away
  • warn me that the code I am looking at is not what was run. provide link to show the diff

Bonus points

  • show links/diffs to other runs in the same build that also have gnucash failures to proof it's always the same
  • links/diffs to gnucash failures in other job groups. I suppose in this case the problem was spotted in TW already
  • single click solution to restart all affected tests at once

Related issues

Related to openQA Project - coordination #19720: [epic] Simplify investigation of job failuresBlocked2019-12-17

History

#1 Updated by lnussel over 2 years ago

  • Project changed from openQA Tests to openQA Project
  • Description updated (diff)
  • Category changed from Bugs in existing tests to 124

#2 Updated by coolo over 2 years ago

  • Subject changed from test fails in gnucash to Make reviewing results easier

#3 Updated by coolo over 2 years ago

  • Subject changed from Make reviewing results easier to [EPIC] Make reviewing results easier
  • Difficulty set to hard

#4 Updated by okurz over 1 year ago

  • Category changed from 124 to Feature requests

#5 Updated by okurz about 1 year ago

#6 Updated by okurz 11 months ago

  • Subject changed from [EPIC] Make reviewing results easier to [epic] Make reviewing results easier

#7 Updated by okurz 6 months ago

  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version set to Ready

waiting for results of #19720 first

#8 Updated by szarate 3 months ago

  • Tracker changed from action to coordination
  • Status changed from Blocked to New
  • Difficulty deleted (hard)

#9 Updated by szarate 3 months ago

#10 Updated by okurz 3 months ago

  • Status changed from New to Blocked
  • Difficulty set to hard

#11 Updated by okurz 3 months ago

  • Parent task set to #39719

Also available in: Atom PDF