action #71536
closedUnhandled perl warnings in t/ui/15-admin-workers.t, not failing tests as expected (possibly other test modules as well)
Description
Observation¶
t/ui/15-admin-workers.t shows perl warnings but still succeeds, e.g.:
$ prove -l t/ui/15-admin-workers.t
t/ui/15-admin-workers.t .. 1/? Use of uninitialized value in pattern match (m//) at /home/okurz/local/os-autoinst/openQA/lib/OpenQA/WebAPI/Controller/Admin/Workers.pm line 28.
Use of uninitialized value in pattern match (m//) at /home/okurz/local/os-autoinst/openQA/lib/OpenQA/WebAPI/Controller/Admin/Workers.pm line 28.
Use of uninitialized value in pattern match (m//) at /home/okurz/local/os-autoinst/openQA/lib/OpenQA/WebAPI/Controller/Admin/Workers.pm line 28.
t/ui/15-admin-workers.t .. 7/? Use of uninitialized value in pattern match (m//) at /home/okurz/local/os-autoinst/openQA/lib/OpenQA/WebAPI/Controller/Admin/Workers.pm line 28.
t/ui/15-admin-workers.t .. ok
All tests successful.
Files=1, Tests=19, 13 wallclock secs ( 0.04 usr 0.00 sys + 1.98 cusr 0.22 csys = 2.24 CPU)
Result: PASS
This is also visible in circleci even though even less obvious. https://app.circleci.com/pipelines/github/os-autoinst/openQA/4264/workflows/c03c8397-a61e-481c-afbb-8c8839a69e9a/jobs/40889/steps is green and there is no warning visible in the log output however https://app.circleci.com/pipelines/github/os-autoinst/openQA/4264/workflows/c03c8397-a61e-481c-afbb-8c8839a69e9a/jobs/40889/artifacts references https://40889-20883829-gh.circle-artifacts.com/0/artifacts/ui/15-admin-workers.t which shows the same warnings as above.
Our test code uses use Test::Warnings ':report_warnings'; but to no avail. I also crosschecked if maybe Test::Warnings in a newer version is the problem itself but the problem reproduces with perl-Test-Warnings-0.026
Expected result¶
- Tests should fail if there is any warning, regardless of the source
Problem¶
This could be related to the fact that we start the SeleniumDriver in a background process and the warning might come from there. Introducing an artificial warning in the main thread is correctly handled.
Suggestions¶
- Try to narrow down the source of the problem and identify from which process the warning comes
- Extend the existing approach to handle warnings or find a different approach for all background processes, e.g. compare to https://github.com/os-autoinst/os-autoinst/pull/1497
Updated by mkittler over 3 years ago
- Status changed from Workable to In Progress
PR for fixing the warning itself: https://github.com/os-autoinst/openQA/pull/3422
Try to narrow down the source of the problem and identify from which process the warning comes
Here the warning comes indeed from a sub process. In this case it is the web UI but we generally wouldn't catch warnings in any sub processes (which are usually spawned by the test helpers create_webapi, create_websocket_server, create_scheduler and create_live_view_handler).
Extend the existing approach to handle warnings …
That would be possible, see https://github.com/os-autoinst/openQA/pull/3423 for a draft. However, it isn't a clear improvement so far (see the PR description) so I'm still trying to improve it.
Updated by mkittler over 3 years ago
- Status changed from In Progress to Resolved
I've improved the PR. It still isn't optimal (see https://github.com/os-autoinst/openQA/pull/3423#issuecomment-699896044) but at least warnings will be catched now in sub processes and the main test will fail and stop if that's the case.