Project

General

Profile

Actions
Copy link

action #178117

closed

coordination #127031: [saga][epic] openQA for SUSE customers

coordination #165393: [epic] Improved code coverage in openQA

[openQA][CI] Prevent unhandled output in prove test calls with automatic checks size:S

Added by okurz 3 months ago. Updated 27 days ago.

Status:
Resolved
Priority:
Low
Assignee:
gpuliti
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2025-03-01
Due date:
% Done:

0%

Estimated time:
Tags:
openQA

Description

Motivation

https://app.circleci.com/pipelines/gh/os-autoinst/openQA/16015/workflows/93b25d9c-129a-40d0-8dec-82b2390fb56b/jobs/154230 shows a problem of unhandled output. We want to have a clean TAP output with all output from tested application code to be captured:

[23:50:36] t/05-scheduler-restart-and-duplicate.t .................... 22/?     # {
    #   '100006' => 100012,
    #   '100008' => 100013,
    #   '100009' => 100014,
    #   '100010' => 100015
    # }
[23:50:36] t/05-scheduler-restart-and-duplicate.t .................... ok    14366 ms ( 0.04 usr  0.01 sys + 13.61 cusr  0.37 csys = 14.03 CPU)
[

Acceptance criteria

  • AC1: Unhandled output from any openQA test called with prove is prevented by automatic checks
  • AC2: Other related projects have according improvements planned or implemented
  • AC3: Output of the tests should still be immediate and not printed with delay

Suggestions

  • Research how others might achieve it already or use our own check of logs or a "prove wrapper"
  • After that works for openQA fix it or report in tickets for related projects
  • Provide a regex representing a line of valid prove output to verify "no output"

Related issues 3 (0 open — 3 closed)

Copied from openQA Project (public) - action #178111: [beginner][openQA][CI] Unhandled output in t/05-scheduler-restart-and-duplicate.tResolvedtinita

Actions
Copied to openQA Project (public) - action #181358: [openQA][CI] Prevent unhandled output in prove test calls with automatic checks in os-autoinstResolvedgpuliti2025-03-01

Actions
Copied to openQA Project (public) - action #181760: [sporadic][openQA][CI] unhandled output in t/full-stack.t "Use of uninitialized value in numeric ne (!=) lib/OpenQA/Worker/Job.pm line 770"Resolvedmkittler

Actions
Actions
Copy link
 #1

Updated by okurz 3 months ago

  • Copied from action #178111: [beginner][openQA][CI] Unhandled output in t/05-scheduler-restart-and-duplicate.t added
Actions
Copy link
 #2

Updated by okurz 3 months ago

  • Target version changed from Ready to Tools - Next
Actions
Copy link
 #3

Updated by okurz about 2 months ago

  • Target version changed from Tools - Next to Ready
Actions
Copy link
 #4

Updated by livdywan about 2 months ago

  • Subject changed from [openQA][CI] Prevent unhandled output in prove test calls with automatic checks to [openQA][CI] Prevent unhandled output in prove test calls with automatic checks size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #5

Updated by gpuliti about 2 months ago

  • Assignee set to gpuliti
Actions
Copy link
 #6

Updated by gpuliti about 2 months ago

  • Status changed from Workable to In Progress
Actions
Copy link
 #7

Updated by gpuliti about 2 months ago

  • Status changed from In Progress to Feedback

There were an explain in the tests that generate a dump of cloned jobs, I've open a pr https://github.com/os-autoinst/openQA/pull/6379

Actions
Copy link
 #8

Updated by livdywan about 2 months ago

gpuliti wrote in #note-7:

There were an explain in the tests that generate a dump of cloned jobs, I've open a pr https://github.com/os-autoinst/openQA/pull/6379

How did you find this? Did you come up with a regex as suggested?

Actions
Copy link
 #9

Updated by tinita about 2 months ago

I made AC1 clearer, that it is not about one single occurrence of unhandled output.

Actions
Copy link
 #10

Updated by tinita about 2 months ago

  • Description updated (diff)

Actually now I made the description clearer. Before it got lost because of conflicting issue commenting.

Actions
Copy link
 #11

Updated by gpuliti about 2 months ago

  • Status changed from Feedback to In Progress

Thanks for the feedback!

Actions
Copy link
 #12

Updated by gpuliti about 2 months ago

I'm writing a wrapper for prove in order to extract from the makefile itself, this is the pr https://github.com/os-autoinst/openQA/pull/6382 I'm open for discussion.

Actions
Copy link
 #13

Updated by gpuliti about 1 month ago

  • Description updated (diff)
Actions
Copy link
 #14

Updated by gpuliti about 1 month ago · Edited

I've rework the script in order to give a better presentation of the unhandled output something like this:

not ok - unhandled output found
  not ok - unhandled output in t/43-cli-api.t
    #{
    #  '100006' => 100012,
    #  '100008' => 100013,
    #  '100009' => 100014,
    #  '100010' => 100015
    #}

It's working locally, but for some reasons not in circleCI, could it be related to the environment?

I've create a commit that need to be removed once the pr is ready that print unhandled output in the log, but unexpectedly in circleCI it's not showing as unhandled and also is completely remove from the output https://app.circleci.com/pipelines/github/os-autoinst/openQA/16707/workflows/d817cf02-1f6c-49ce-8e79-acb140212a03/jobs/162072

I think is related to this part of the code https://github.com/os-autoinst/openQA/pull/6382/files#diff-bc829f3ebe1d7706717660707b70f6b74bdc88dbf2428ddd6728125882b8d96eR38-R41 and I'm investigating around it. No, it's not. I'm blocking the ticket to request assistance!

Actions
Copy link
 #15

Updated by gpuliti about 1 month ago

  • Status changed from In Progress to Blocked
Actions
Copy link
 #16

Updated by livdywan about 1 month ago

  • Status changed from Blocked to Workable

Please keep in mind that Blocked is for external dependencies where we have a ticket to block on, not for open questions you need help answering

Actions
Copy link
 #17

Updated by gpuliti about 1 month ago

  • Status changed from Workable to Feedback
Actions
Copy link
 #18

Updated by gpuliti about 1 month ago

  • Status changed from Feedback to In Progress
Actions
Copy link
 #19

Updated by gpuliti about 1 month ago

  • Status changed from In Progress to Feedback

Cover the unhandled output with https://github.com/os-autoinst/openQA/pull/6397

After the pr#6397 has been merged I'll rebase https://github.com/os-autoinst/openQA/pull/6382 wait for reviews and if accepted I'll resolve the ticket.

Actions
Copy link
 #20

Updated by gpuliti about 1 month ago · Edited

  • Status changed from Feedback to Resolved
Actions
Copy link
 #22

Updated by tinita about 1 month ago

  • Status changed from Resolved to In Progress
Actions
Copy link
 #23

Updated by tinita about 1 month ago

Also what about AC2?

Actions
Copy link
 #24

Updated by gpuliti about 1 month ago

I think I got confused between pr, sorry!

About the AC2, I've take a look into os-autoinst/os-autoinst, but it's more complex to work with and edit like the way I do for os-autoinst/openqa.

Actions
Copy link
 #25

Updated by gpuliti about 1 month ago

  • Description updated (diff)

As discussed during the dev daily today I'll move the AC2 into a new ticket because it is not feasible to complete it today due to online meetings.

Actions
Copy link
 #26

Updated by gpuliti about 1 month ago

  • Description updated (diff)
Actions
Copy link
 #27

Updated by gpuliti about 1 month ago

  • Description updated (diff)
Actions
Copy link
 #28

Updated by gpuliti about 1 month ago

  • Copied to action #181358: [openQA][CI] Prevent unhandled output in prove test calls with automatic checks in os-autoinst added
Actions
Copy link
 #29

Updated by gpuliti about 1 month ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #30

Updated by gpuliti 27 days ago

  • Status changed from Feedback to Resolved

Since AC2 is moved into a new ticket and the pr https://github.com/os-autoinst/openQA/pull/6382 is now merged, I set the ticket as resolved.

Actions
Copy link
 #31

Updated by okurz 21 days ago

  • Copied to action #181760: [sporadic][openQA][CI] unhandled output in t/full-stack.t "Use of uninitialized value in numeric ne (!=) lib/OpenQA/Worker/Job.pm line 770" added
Actions
Copy link

Also available in: Atom PDF