Project

General

Profile

Actions

action #178117

closed

coordination #127031: [saga][epic] openQA for SUSE customers

coordination #165393: [epic] Improved code coverage in openQA

[openQA][CI] Prevent unhandled output in prove test calls with automatic checks size:S

Added by okurz about 2 months ago. Updated 6 minutes ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2025-03-01
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Motivation

https://app.circleci.com/pipelines/gh/os-autoinst/openQA/16015/workflows/93b25d9c-129a-40d0-8dec-82b2390fb56b/jobs/154230 shows a problem of unhandled output. We want to have a clean TAP output with all output from tested application code to be captured:

[23:50:36] t/05-scheduler-restart-and-duplicate.t .................... 22/?     # {
    #   '100006' => 100012,
    #   '100008' => 100013,
    #   '100009' => 100014,
    #   '100010' => 100015
    # }
[23:50:36] t/05-scheduler-restart-and-duplicate.t .................... ok    14366 ms ( 0.04 usr  0.01 sys + 13.61 cusr  0.37 csys = 14.03 CPU)
[

Acceptance criteria

  • AC1: Unhandled output from any openQA test called with prove is prevented by automatic checks
  • AC2: Other related projects have according improvements planned or implemented
  • AC3: Output of the tests should still be immediate and not printed with delay

Suggestions

  • Research how others might achieve it already or use our own check of logs or a "prove wrapper"
  • After that works for openQA fix it or report in tickets for related projects
  • Provide a regex representing a line of valid prove output to verify "no output"

Related issues 1 (0 open1 closed)

Copied from openQA Project (public) - action #178111: [beginner][openQA][CI] Unhandled output in t/05-scheduler-restart-and-duplicate.tResolvedtinita

Actions
Actions #1

Updated by okurz about 2 months ago

  • Copied from action #178111: [beginner][openQA][CI] Unhandled output in t/05-scheduler-restart-and-duplicate.t added
Actions #2

Updated by okurz about 2 months ago

  • Target version changed from Ready to Tools - Next
Actions #3

Updated by okurz 27 days ago

  • Target version changed from Tools - Next to Ready
Actions #4

Updated by livdywan 20 days ago

  • Subject changed from [openQA][CI] Prevent unhandled output in prove test calls with automatic checks to [openQA][CI] Prevent unhandled output in prove test calls with automatic checks size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #5

Updated by gpuliti 16 days ago

  • Assignee set to gpuliti
Actions #6

Updated by gpuliti 16 days ago

  • Status changed from Workable to In Progress
Actions #7

Updated by gpuliti 15 days ago

  • Status changed from In Progress to Feedback

There were an explain in the tests that generate a dump of cloned jobs, I've open a pr https://github.com/os-autoinst/openQA/pull/6379

Actions #8

Updated by livdywan 15 days ago

gpuliti wrote in #note-7:

There were an explain in the tests that generate a dump of cloned jobs, I've open a pr https://github.com/os-autoinst/openQA/pull/6379

How did you find this? Did you come up with a regex as suggested?

Actions #9

Updated by tinita 15 days ago

I made AC1 clearer, that it is not about one single occurrence of unhandled output.

Actions #10

Updated by tinita 15 days ago

  • Description updated (diff)

Actually now I made the description clearer. Before it got lost because of conflicting issue commenting.

Actions #11

Updated by gpuliti 14 days ago

  • Status changed from Feedback to In Progress

Thanks for the feedback!

Actions #12

Updated by gpuliti 14 days ago

I'm writing a wrapper for prove in order to extract from the makefile itself, this is the pr https://github.com/os-autoinst/openQA/pull/6382 I'm open for discussion.

Actions #13

Updated by gpuliti 12 days ago

  • Description updated (diff)
Actions #14

Updated by gpuliti 7 days ago ยท Edited

I've rework the script in order to give a better presentation of the unhandled output something like this:

not ok - unhandled output found
  not ok - unhandled output in t/43-cli-api.t
    #{
    #  '100006' => 100012,
    #  '100008' => 100013,
    #  '100009' => 100014,
    #  '100010' => 100015
    #}

It's working locally, but for some reasons not in circleCI, could it be related to the environment?

I've create a commit that need to be removed once the pr is ready that print unhandled output in the log, but unexpectedly in circleCI it's not showing as unhandled and also is completely remove from the output https://app.circleci.com/pipelines/github/os-autoinst/openQA/16707/workflows/d817cf02-1f6c-49ce-8e79-acb140212a03/jobs/162072

I think is related to this part of the code https://github.com/os-autoinst/openQA/pull/6382/files#diff-bc829f3ebe1d7706717660707b70f6b74bdc88dbf2428ddd6728125882b8d96eR38-R41 and I'm investigating around it. No, it's not. I'm blocking the ticket to request assistance!

Actions #15

Updated by gpuliti 7 days ago

  • Status changed from In Progress to Blocked
Actions #16

Updated by livdywan 7 days ago

  • Status changed from Blocked to Workable

Please keep in mind that Blocked is for external dependencies where we have a ticket to block on, not for open questions you need help answering

Actions #17

Updated by gpuliti 7 days ago

  • Status changed from Workable to Feedback
Actions #18

Updated by gpuliti 7 days ago

  • Status changed from Feedback to In Progress
Actions #19

Updated by gpuliti 6 days ago

  • Status changed from In Progress to Feedback

Cover the unhandled output with https://github.com/os-autoinst/openQA/pull/6397

After the pr#6397 has been merged I'll rebase https://github.com/os-autoinst/openQA/pull/6382 wait for reviews and if accepted I'll resolve the ticket.

Actions #20

Updated by gpuliti 6 minutes ago

  • Status changed from Feedback to Resolved

Merged!

Actions

Also available in: Atom PDF