Be more robust about spurious errors
As a tester reviewing failed tests in flaky environments (involving network, timeouts, reviewing webpages) we want flaky tests to retry automatically to not cause false positives in tests
- a test with spurious errors that is normally restarted by hand is restarted automatically
- the retry is reported as a "soft fail" (or another state not "fail" or "passed") instead of a "fail" as it would be without this change
- the "retry" behaviour should be as low level as possible to save testing time but still be able to report the retry as a "soft fail"
- For every job that does not have a label or bugref, retrigger some times to see if it's sporadic within the same scenario. Like rescheduling on incomplete but on failed
see irc conversation:
<okurz> ancor: wait, shouldn't we try to reproduce it locally, maybe? <ancor> okurz: I'm taking a look to the logs now, but it wouldn't be the first spurious error observed in openQA <okurz> ancor: yes, I know but I like spurious errors to be better handled instead of just "let's retry, waste some time/build cycles, and see if it happens again" :-) <ancor> okurz: I have been usually told than CPU cycles are cheaper than developer ones :-) <ancor> machines don't get bored :-) <okurz> ancor: of course you can restart but let's take it as good intentions for next year to handle spurious errors better :-) of course build cycles are cheaper but I am thinking about a better automatic spurious error detection, e.g. "retry if canditate for spurious" <okurz> ancor: would waste even more build cycles but actually save more develper hours <okurz> ancor: so I would actually like to optimize the time we need to detect spurious errors ... <ancor> okurz: anyway, looking at the logs. It really looks bad "Subprocess failed. Error: RPM failed: error: rpmdb: fsync: Read-only file system" ...
#4 Updated by okurz over 1 year ago
- Checklist deleted (
[ ] SLE, [ ] Leap, [ ] TW)
- Project changed from openQA Tests to openQA Project
- Subject changed from [tools] Be more robust about spurious errors to Be more robust about spurious errors
- Category changed from Enhancement to existing tests to Feature requests
- Priority changed from Normal to Low
- Target version set to future