Project

General

Profile

Actions
Copy link

action #178117

open

coordination #127031: [saga][epic] openQA for SUSE customers

coordination #165393: [epic] Improved code coverage in openQA

[openQA][CI] Prevent unhandled output in prove test calls with automatic checks size:S

Added by okurz about 2 months ago. Updated about 23 hours ago.

Status:
Feedback
Priority:
Low
Assignee:
gpuliti
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2025-03-01
Due date:
% Done:

0%

Estimated time:
Tags:
openQA

Description

Motivation

https://app.circleci.com/pipelines/gh/os-autoinst/openQA/16015/workflows/93b25d9c-129a-40d0-8dec-82b2390fb56b/jobs/154230 shows a problem of unhandled output. We want to have a clean TAP output with all output from tested application code to be captured:

[23:50:36] t/05-scheduler-restart-and-duplicate.t .................... 22/?     # {
    #   '100006' => 100012,
    #   '100008' => 100013,
    #   '100009' => 100014,
    #   '100010' => 100015
    # }
[23:50:36] t/05-scheduler-restart-and-duplicate.t .................... ok    14366 ms ( 0.04 usr  0.01 sys + 13.61 cusr  0.37 csys = 14.03 CPU)
[

Acceptance criteria

  • AC1: Unhandled output from any openQA test called with prove is prevented by automatic checks
  • AC2: Other related projects have according improvements planned or implemented
  • AC3: Output of the tests should still be immediate and not printed with delay

Suggestions

  • Research how others might achieve it already or use our own check of logs or a "prove wrapper"
  • After that works for openQA fix it or report in tickets for related projects
  • Provide a regex representing a line of valid prove output to verify "no output"

Related issues 1 (0 open — 1 closed)

Copied from openQA Project (public) - action #178111: [beginner][openQA][CI] Unhandled output in t/05-scheduler-restart-and-duplicate.tResolvedtinita

Actions
Actions
Copy link
 #1

Updated by okurz about 2 months ago

  • Copied from action #178111: [beginner][openQA][CI] Unhandled output in t/05-scheduler-restart-and-duplicate.t added
Actions
Copy link
 #2

Updated by okurz about 1 month ago

  • Target version changed from Ready to Tools - Next
Actions
Copy link
 #3

Updated by okurz 22 days ago

  • Target version changed from Tools - Next to Ready
Actions
Copy link
 #4

Updated by livdywan 15 days ago

  • Subject changed from [openQA][CI] Prevent unhandled output in prove test calls with automatic checks to [openQA][CI] Prevent unhandled output in prove test calls with automatic checks size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #5

Updated by gpuliti 11 days ago

  • Assignee set to gpuliti
Actions
Copy link
 #6

Updated by gpuliti 11 days ago

  • Status changed from Workable to In Progress
Actions
Copy link
 #7

Updated by gpuliti 10 days ago

  • Status changed from In Progress to Feedback

There were an explain in the tests that generate a dump of cloned jobs, I've open a pr https://github.com/os-autoinst/openQA/pull/6379

Actions
Copy link
 #8

Updated by livdywan 10 days ago

gpuliti wrote in #note-7:

There were an explain in the tests that generate a dump of cloned jobs, I've open a pr https://github.com/os-autoinst/openQA/pull/6379

How did you find this? Did you come up with a regex as suggested?

Actions
Copy link
 #9

Updated by tinita 10 days ago

I made AC1 clearer, that it is not about one single occurrence of unhandled output.

Actions
Copy link
 #10

Updated by tinita 10 days ago

  • Description updated (diff)

Actually now I made the description clearer. Before it got lost because of conflicting issue commenting.

Actions
Copy link
 #11

Updated by gpuliti 9 days ago

  • Status changed from Feedback to In Progress

Thanks for the feedback!

Actions
Copy link
 #12

Updated by gpuliti 9 days ago

I'm writing a wrapper for prove in order to extract from the makefile itself, this is the pr https://github.com/os-autoinst/openQA/pull/6382 I'm open for discussion.

Actions
Copy link
 #13

Updated by gpuliti 7 days ago

  • Description updated (diff)
Actions
Copy link
 #14

Updated by gpuliti 2 days ago ยท Edited

I've rework the script in order to give a better presentation of the unhandled output something like this:

not ok - unhandled output found
  not ok - unhandled output in t/43-cli-api.t
    #{
    #  '100006' => 100012,
    #  '100008' => 100013,
    #  '100009' => 100014,
    #  '100010' => 100015
    #}

It's working locally, but for some reasons not in circleCI, could it be related to the environment?

I've create a commit that need to be removed once the pr is ready that print unhandled output in the log, but unexpectedly in circleCI it's not showing as unhandled and also is completely remove from the output https://app.circleci.com/pipelines/github/os-autoinst/openQA/16707/workflows/d817cf02-1f6c-49ce-8e79-acb140212a03/jobs/162072

I think is related to this part of the code https://github.com/os-autoinst/openQA/pull/6382/files#diff-bc829f3ebe1d7706717660707b70f6b74bdc88dbf2428ddd6728125882b8d96eR38-R41 and I'm investigating around it. No, it's not. I'm blocking the ticket to request assistance!

Actions
Copy link
 #15

Updated by gpuliti 2 days ago

  • Status changed from In Progress to Blocked
Actions
Copy link
 #16

Updated by livdywan 2 days ago

  • Status changed from Blocked to Workable

Please keep in mind that Blocked is for external dependencies where we have a ticket to block on, not for open questions you need help answering

Actions
Copy link
 #17

Updated by gpuliti 2 days ago

  • Status changed from Workable to Feedback
Actions
Copy link
 #18

Updated by gpuliti 2 days ago

  • Status changed from Feedback to In Progress
Actions
Copy link
 #19

Updated by gpuliti about 23 hours ago

  • Status changed from In Progress to Feedback

Cover the unhandled output with https://github.com/os-autoinst/openQA/pull/6397

After the pr#6397 has been merged I'll rebase https://github.com/os-autoinst/openQA/pull/6382 wait for reviews and if accepted I'll resolve the ticket.

Actions
Copy link

Also available in: Atom PDF