QA » openQA Project

IMO a generic "pause on module failure" option might be useful here and get to at least 80% of the goal

For needles, retrying the match makes sense as it's possible to create new needles and using dev mode + VNC connection also possible to change the system state. Retrying an assert_script_run however does not allow to change the command or expected outcome, just the system state. In my experience, the main reason of script failures is that the command itself is incorrect, which can't be fixed by simply retrying.

This new option could be implemented by creating a new lower level pause($reason) backend command and just send that on module failure (but before the post_fail_hook) if "pause on module failure" is enabled, or a module_failure($module) to have the backend decide what to do.

favogt wrote:

IMO a generic "pause on module failure" option might be useful here and get to at least 80% of the goal

Yes, as long as there will be an option to continue, instead of following "normal" failure action (post_fail_hook, reverting to snapshot, aborting all together etc).

This new option could be implemented by creating a new lower level pause($reason) backend command and just send that on module failure (but before the post_fail_hook) if "pause on module failure" is enabled, or a module_failure($module) to have the backend decide what to do.

Having primitive like this, I could also extend distribution script_run (or other commands) to add pause points.

I concur with @fvogt, a generic "pause on module failure" option makes most sense. We don't want to have specific handling for several reasons the test can fail in the future. Of course when continuing there should be an option to ignore the failure or to re-do the failing command.

This new option could be implemented by creating a new lower level pause($reason) backend command and just send that on module failure (but before the post_fail_hook) if "pause on module failure" is enabled, or a module_failure($module) to have the backend decide what to do.

Out of my head I'd say that sounds like a good way of implementing it, indeed. Since @okurz has added it to the backlog I'd like to work on it next week (unless someone else grabs the ticket of course).

Thanks for working on this!

mkittler wrote:

• Being able to resume and continue where the test execution has left off is a bit of a problem. When an exception has been thrown the control flow is intercepted and there's no way to jump back (or is there a Perl hack I'm unaware of?). So if one resumes after an arbitrary unhandled exception the execution will simply continue on the next test module.
• In case the test fails via a failing check using a test API function it would be possible to avoid throwing the exception. This likely means adjusting many test API functions but it should be doable (as shown in the draft PR for assert_script_run).

I think this is perfectly fine - do the pause in modified test functions. In some of them it wouldn't really make sense, for example continuing from script_output with no result wouldn't work in most cases anyway.
For me personally, just modifying assert_script_run, and providing alternative "test fail" function (to use instead of die when continuation make sense) would be enough.

Draft PR: https://github.com/os-autoinst/os-autoinst/pull/2198

How I can test this? Is there related frontend change? Especially, how to choose whether to ignore the failure or not from developer mode? IIUC I can enable the feature with PAUSE_ON_FAILURE=1, but can't choose how to continue.

I'm working on this low prio ticket on the side. So I'm not stuck. At this point there's nothing for you to test because I'd like to test it on my own a little bit more first.

However, I could already come up with a few improvements for the developer mode on the way: https://github.com/os-autoinst/os-autoinst/pull/2204, https://github.com/os-autoinst/openQA/pull/4897

• I've just updated the PR and also created one for UI changed in openQA: https://github.com/os-autoinst/openQA/pull/4906
• For the question about testing, see my comment on GitHub: https://github.com/os-autoinst/os-autoinst/pull/2198#issuecomment-1318788214

Thanks! I tested it both for ordinary "pause on failure", as well as pausing on assert_script_run failure and ignoring that failure (with both above PRs included for that) and it works great.

AC2: marmarek is happy

I'm happy :)

Great :-)

Cool, so everybody is happy :)

Waiting for outstanding PR https://github.com/os-autoinst/os-autoinst/pull/2225