action #178117
closedcoordination #127031: [saga][epic] openQA for SUSE customers
coordination #165393: [epic] Improved code coverage in openQA
[openQA][CI] Prevent unhandled output in prove test calls with automatic checks size:S
0%
Description
Motivation¶
https://app.circleci.com/pipelines/gh/os-autoinst/openQA/16015/workflows/93b25d9c-129a-40d0-8dec-82b2390fb56b/jobs/154230 shows a problem of unhandled output. We want to have a clean TAP output with all output from tested application code to be captured:
[23:50:36] t/05-scheduler-restart-and-duplicate.t .................... 22/? # {
# '100006' => 100012,
# '100008' => 100013,
# '100009' => 100014,
# '100010' => 100015
# }
[23:50:36] t/05-scheduler-restart-and-duplicate.t .................... ok 14366 ms ( 0.04 usr 0.01 sys + 13.61 cusr 0.37 csys = 14.03 CPU)
[
Acceptance criteria¶
- AC1: Unhandled output from any openQA test called with prove is prevented by automatic checks
- AC2: Other related projects have according improvements planned or implemented
- AC3: Output of the tests should still be immediate and not printed with delay
Suggestions¶
- Research how others might achieve it already or use our own check of logs or a "prove wrapper"
- After that works for openQA fix it or report in tickets for related projects
- Provide a regex representing a line of valid prove output to verify "no output"
Updated by okurz about 2 months ago
- Copied from action #178111: [beginner][openQA][CI] Unhandled output in t/05-scheduler-restart-and-duplicate.t added
Updated by okurz about 2 months ago
- Target version changed from Ready to Tools - Next
Updated by gpuliti 15 days ago
- Status changed from In Progress to Feedback
There were an explain in the tests that generate a dump of cloned jobs, I've open a pr https://github.com/os-autoinst/openQA/pull/6379
Updated by gpuliti 14 days ago
I'm writing a wrapper for prove in order to extract from the makefile itself, this is the pr https://github.com/os-autoinst/openQA/pull/6382 I'm open for discussion.
Updated by gpuliti 7 days ago ยท Edited
I've rework the script in order to give a better presentation of the unhandled output something like this:
not ok - unhandled output found
not ok - unhandled output in t/43-cli-api.t
#{
# '100006' => 100012,
# '100008' => 100013,
# '100009' => 100014,
# '100010' => 100015
#}
It's working locally, but for some reasons not in circleCI, could it be related to the environment?
I've create a commit that need to be removed once the pr is ready that print unhandled output in the log, but unexpectedly in circleCI it's not showing as unhandled and also is completely remove from the output https://app.circleci.com/pipelines/github/os-autoinst/openQA/16707/workflows/d817cf02-1f6c-49ce-8e79-acb140212a03/jobs/162072
I think is related to this part of the code https://github.com/os-autoinst/openQA/pull/6382/files#diff-bc829f3ebe1d7706717660707b70f6b74bdc88dbf2428ddd6728125882b8d96eR38-R41 and I'm investigating around it. No, it's not. I'm blocking the ticket to request assistance!
Updated by gpuliti 6 days ago
- Status changed from In Progress to Feedback
Cover the unhandled output with https://github.com/os-autoinst/openQA/pull/6397
After the pr#6397 has been merged I'll rebase https://github.com/os-autoinst/openQA/pull/6382 wait for reviews and if accepted I'll resolve the ticket.