Improve automatic carryover to be more strict - when bugs relate to individual steps within test details
Automatic carryover is cool feature which ease test reviewer life a lot, but because we not yet invent AI for openQA sometimes automatic carryover label jobs which actually need human eye.
Acceptance criteria for this ticket would be - dramatically decrease cases when automatic algorithm label jobs which should not be labeled or fully exclude such cases.
Examples of Negative cases :
- https://openqa.suse.de/tests/938443 ( false carryover of fixed bug , in a fact test regression )
- https://openqa.suse.de/tests/938555 ( false carryover of fixed bug , in a fact new bug)
- https://openqa.suse.de/tests/937960 ( false carryover same module fail but actually another bug reproduces)
- https://openqa.suse.de/tests/938212 ( false carryover same module fail but actually another bug reproduces)
#2 Updated by okurz about 4 years ago
- Subject changed from [tools] Negative cases of automatic carryover to [tools] Improve automatic carryover to be more strict
- Category changed from Support to Feature requests
- Assignee deleted (
- Priority changed from Normal to Low
In the examples you mentioned carry-over works as expected but is carrying over issues which are not related because they refer to different steps within that job. A way to remedy that from the side of tests is to split the tests into more modules. A way to improve that from the side of openQA itself might be to configure the automatic carry over to look at steps and not only modules, maybe paired with some fuzziness, e.g. if still fails in step N+-1.
#7 Updated by asmorodskyi about 4 years ago
another approach - can we somehow detect reason of failure ? I mean when we writing into log "Test died in ..." ? We can standardize this into some structure like :
failure_reason : needle match / script run failed / general error etc.
failure_message : will contain message like "Needle match failed" / "Script run timed out " / "Script end with error"
and openQA can compare this between jobs
#9 Updated by asmorodskyi about 4 years ago
https://openqa.suse.de/tests/1036627#step/force_cron_run/4 - another example of false carryover . test fails before it reach described in bug failure point.