Actions
coordination #9966
opencoordination #102915: [saga][epic] Automated classification of failures
[epic] Be more robust about spurious errors
Start date:
2015-12-18
Due date:
% Done:
0%
Estimated time:
Description
user story¶
As a tester reviewing failed tests in flaky environments (involving network, timeouts, reviewing webpages) we want flaky tests to retry automatically to not cause false positives in tests
acceptance criteria¶
- a test with spurious errors that is normally restarted by hand is restarted automatically
- the retry is reported as a "soft fail" (or another state not "fail" or "passed") instead of a "fail" as it would be without this change
implementation ideas¶
- the "retry" behaviour should be as low level as possible to save testing time but still be able to report the retry as a "soft fail"
- For every job that does not have a label or bugref, retrigger some times to see if it's sporadic within the same scenario. Like rescheduling on incomplete but on failed
further details¶
reasoning¶
see irc conversation:
<okurz> ancor: wait, shouldn't we try to reproduce it locally, maybe?
<ancor> okurz: I'm taking a look to the logs now, but it wouldn't be the first spurious error observed in openQA
<okurz> ancor: yes, I know but I like spurious errors to be better handled instead of just "let's retry, waste some time/build cycles, and see if it happens again" :-)
<ancor> okurz: I have been usually told than CPU cycles are cheaper than developer ones :-)
<ancor> machines don't get bored :-)
<okurz> ancor: of course you can restart but let's take it as good intentions for next year to handle spurious errors better :-) of course build cycles are cheaper but I am thinking about a better automatic spurious error detection, e.g. "retry if canditate for spurious"
<okurz> ancor: would waste even more build cycles but actually save more develper hours
<okurz> ancor: so I would actually like to optimize the time we need to detect spurious errors
...
<ancor> okurz: anyway, looking at the logs. It really looks bad "Subprocess failed. Error: RPM failed: error: rpmdb: fsync: Read-only file system"
...
Updated by RBrownSUSE about 9 years ago
- Checklist item changed from to [ ] SLE, [ ] Leap, [ ] TW
Updated by okurz about 8 years ago
- Category set to Enhancement to existing tests
Updated by asmorodskyi over 7 years ago
- Subject changed from Be more robust about spurious errors to [tools] Be more robust about spurious errors
Updated by okurz about 5 years ago
- Checklist item changed from [ ] SLE, [ ] Leap, [ ] TW to
- Project changed from openQA Tests (public) to openQA Project (public)
- Subject changed from [tools] Be more robust about spurious errors to Be more robust about spurious errors
- Category changed from Enhancement to existing tests to Feature requests
- Priority changed from Normal to Low
- Target version set to future
Updated by okurz about 5 years ago
- Related to action #13242: WDYT: For every job that does not have a label or bugref, retrigger some times to see if it's sporadic. Like rescheduling on incomplete but on failed added
Updated by okurz 11 months ago
- Copied to action #155731: [brainstorm] Be more robust about spurious errors added
Actions