QA » openQA Project

a 'step' in this context is a bit fuzzy defined - if you do a loop in the test, there will be more screens and as such more steps.

I would go rather along the line of 'same line of code', but that needs better markup from os-autoinst

what about comparison of failed needle ? does not matter actually which step it is , just show me same picture that was last time and I will be convinced that this is the same failure ?

openQA has no image comparision - next to bit identical. And failures are not always assert_screen

but we have comparison in isotovideo. We can extract it into library and reuse in openQA

another approach - can we somehow detect reason of failure ? I mean when we writing into log "Test died in ..." ? We can standardize this into some structure like :
{
failure_reason : needle match / script run failed / general error etc.
failure_message : will contain message like "Needle match failed" / "Script run timed out " / "Script end with error"
}
and openQA can compare this between jobs

yes, good idea. I recommend that at we make sure that every test failure outputs this "record_info" box. We could check for this then within openQA without needing to move any image comparison algorithm to openQA.

https://openqa.suse.de/tests/1036627#step/force_cron_run/4 - another example of false carryover . test fails before it reach described in bug failure point.