Project

General

Profile

action #71536

Unhandled perl warnings in t/ui/15-admin-workers.t, not failing tests as expected (possibly other test modules as well)

Added by okurz 10 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-09-18
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

t/ui/15-admin-workers.t shows perl warnings but still succeeds, e.g.:

$ prove -l t/ui/15-admin-workers.t 
t/ui/15-admin-workers.t .. 1/? Use of uninitialized value in pattern match (m//) at /home/okurz/local/os-autoinst/openQA/lib/OpenQA/WebAPI/Controller/Admin/Workers.pm line 28.
Use of uninitialized value in pattern match (m//) at /home/okurz/local/os-autoinst/openQA/lib/OpenQA/WebAPI/Controller/Admin/Workers.pm line 28.
Use of uninitialized value in pattern match (m//) at /home/okurz/local/os-autoinst/openQA/lib/OpenQA/WebAPI/Controller/Admin/Workers.pm line 28.
t/ui/15-admin-workers.t .. 7/? Use of uninitialized value in pattern match (m//) at /home/okurz/local/os-autoinst/openQA/lib/OpenQA/WebAPI/Controller/Admin/Workers.pm line 28.
t/ui/15-admin-workers.t .. ok    
All tests successful.
Files=1, Tests=19, 13 wallclock secs ( 0.04 usr  0.00 sys +  1.98 cusr  0.22 csys =  2.24 CPU)
Result: PASS

This is also visible in circleci even though even less obvious. https://app.circleci.com/pipelines/github/os-autoinst/openQA/4264/workflows/c03c8397-a61e-481c-afbb-8c8839a69e9a/jobs/40889/steps is green and there is no warning visible in the log output however https://app.circleci.com/pipelines/github/os-autoinst/openQA/4264/workflows/c03c8397-a61e-481c-afbb-8c8839a69e9a/jobs/40889/artifacts references https://40889-20883829-gh.circle-artifacts.com/0/artifacts/ui/15-admin-workers.t which shows the same warnings as above.

Our test code uses use Test::Warnings ':report_warnings'; but to no avail. I also crosschecked if maybe Test::Warnings in a newer version is the problem itself but the problem reproduces with perl-Test-Warnings-0.026

Expected result

  • Tests should fail if there is any warning, regardless of the source

Problem

This could be related to the fact that we start the SeleniumDriver in a background process and the warning might come from there. Introducing an artificial warning in the main thread is correctly handled.

Suggestions

  • Try to narrow down the source of the problem and identify from which process the warning comes
  • Extend the existing approach to handle warnings or find a different approach for all background processes, e.g. compare to https://github.com/os-autoinst/os-autoinst/pull/1497

History

#1 Updated by mkittler 10 months ago

  • Assignee set to mkittler

#2 Updated by mkittler 10 months ago

  • Status changed from Workable to In Progress

PR for fixing the warning itself: https://github.com/os-autoinst/openQA/pull/3422

Try to narrow down the source of the problem and identify from which process the warning comes

Here the warning comes indeed from a sub process. In this case it is the web UI but we generally wouldn't catch warnings in any sub processes (which are usually spawned by the test helpers create_webapi, create_websocket_server, create_scheduler and create_live_view_handler).

Extend the existing approach to handle warnings …

That would be possible, see https://github.com/os-autoinst/openQA/pull/3423 for a draft. However, it isn't a clear improvement so far (see the PR description) so I'm still trying to improve it.

#3 Updated by mkittler 10 months ago

  • Status changed from In Progress to Resolved

I've improved the PR. It still isn't optimal (see https://github.com/os-autoinst/openQA/pull/3423#issuecomment-699896044) but at least warnings will be catched now in sub processes and the main test will fail and stop if that's the case.

Also available in: Atom PDF