Project

General

Profile

action #9966

Be more robust about spurious errors

Added by okurz almost 5 years ago. Updated 8 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2015-12-18
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

user story

As a tester reviewing failed tests in flaky environments (involving network, timeouts, reviewing webpages) we want flaky tests to retry automatically to not cause false positives in tests

acceptance criteria

  • a test with spurious errors that is normally restarted by hand is restarted automatically
  • the retry is reported as a "soft fail" (or another state not "fail" or "passed") instead of a "fail" as it would be without this change

implementation ideas

  • the "retry" behaviour should be as low level as possible to save testing time but still be able to report the retry as a "soft fail"
  • For every job that does not have a label or bugref, retrigger some times to see if it's sporadic within the same scenario. Like rescheduling on incomplete but on failed

further details

reasoning

see irc conversation:

<okurz> ancor: wait, shouldn't we try to reproduce it locally, maybe?
<ancor> okurz: I'm taking a look to the logs now, but it wouldn't be the first spurious error observed in openQA
<okurz> ancor: yes, I know but I like spurious errors to be better handled instead of just "let's retry, waste some time/build cycles, and see if it happens again" :-)
<ancor> okurz: I have been usually told than CPU cycles are cheaper than developer ones :-)
<ancor> machines don't get bored :-)
<okurz> ancor: of course you can restart but let's take it as good intentions for next year to handle spurious errors better :-) of course build cycles are cheaper but I am thinking about a better automatic spurious error detection, e.g. "retry if canditate for spurious"
<okurz> ancor: would waste even more build cycles but actually save more develper hours
<okurz> ancor: so I would actually like to optimize the time we need to detect spurious errors
...
<ancor> okurz: anyway, looking at the logs. It really looks bad "Subprocess failed. Error: RPM failed: error: rpmdb: fsync: Read-only file system"
...

Related issues

Related to openQA Project - action #13242: WDYT: For every job that does not have a label or bugref, retrigger some times to see if it's sporadic. Like rescheduling on incomplete but on failedRejected2016-11-25

History

#1 Updated by RBrownSUSE over 4 years ago

  • Checklist set to [ ] SLE, [ ] Leap, [ ] TW

#2 Updated by okurz almost 4 years ago

  • Category set to Enhancement to existing tests

#3 Updated by asmorodskyi over 3 years ago

  • Subject changed from Be more robust about spurious errors to [tools] Be more robust about spurious errors

#4 Updated by okurz 8 months ago

  • Checklist deleted ([ ] SLE, [ ] Leap, [ ] TW)
  • Project changed from openQA Tests to openQA Project
  • Subject changed from [tools] Be more robust about spurious errors to Be more robust about spurious errors
  • Category changed from Enhancement to existing tests to Feature requests
  • Priority changed from Normal to Low
  • Target version set to future

#5 Updated by okurz 8 months ago

  • Related to action #13242: WDYT: For every job that does not have a label or bugref, retrigger some times to see if it's sporadic. Like rescheduling on incomplete but on failed added

#6 Updated by okurz 8 months ago

  • Description updated (diff)

Also available in: Atom PDF