action #137651
closed[qem-dashboard] CI tests using Playwright are broken
0%
Description
Observation¶
While working on #132689 i noticed that the Playwright UI tests currently do not work properly and hang for 5 hours before failing with this error:
Run prove -l -v t/*.t.js
Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory
Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.
Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.
Error: The operation was canceled.
Locally the test passes, so this should be GitHub Action specific in some way. But this will block any 3rd party contributions for the time being.
Acceptance criteria¶
- AC1: Tests pass in GitHub Action
Suggestions¶
- Check if there are new best practices for using Playwright in GitHub Actions.
- Check if there are any known IO::Socket::SSL errors that could cause this.
- Maybe Try to SSH into a GitHub Action run.
Updated by tinita about 1 year ago
The blib
in the error message blib/lib/Net/SSLeay.pm
is suspicious. Tests are run with prove -l
, so blib can't come from there.
Updated by kraih about 1 year ago
Also opened a ticket upstream for the deprecation warning, it's harmless in the context of this issue though. https://github.com/jhthorsen/mojolicious-plugin-webpack/issues/16
Updated by livdywan about 1 year ago
kraih wrote in #note-2:
Also opened a ticket upstream for the deprecation warning, it's harmless in the context of this issue though. https://github.com/jhthorsen/mojolicious-plugin-webpack/issues/16
The logs claim that this is the fatal error, though? Supposedly the cover issue is being ignored?
Updated by kraih about 1 year ago
livdywan wrote in #note-3:
The logs claim that this is the fatal error, though? Supposedly the cover issue is being ignored?
Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.
Updated by livdywan about 1 year ago
Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.
Run prove -l -v t/*.t.js
Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory
Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.
Says ignored.
Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.
Says Error.
Error: The operation was canceled.
Updated by kraih about 1 year ago
livdywan wrote in #note-6:
Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.
Run prove -l -v t/*.t.js Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.
Says ignored.
Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.
Says Error.
Error: The operation was canceled.
I'm afraid you've misinterpreted the logs. Error only means that this has been printed to STDERR in the GitHub Action. Pass or failure of the step is decided by the final exit code. This step was cancelled by GitHub because it somehow ended up in an infinite loop. The deprecation is a Perl warning printed to STDERR, and completely harmless here.
Updated by kraih about 1 year ago
I suspect that the real problem is actually something in one of the JavaScript dependencies that have recently been updated.
Updated by kraih about 1 year ago
Downgrading the JavaScript dependencies resolved the problem, lets see if i can find the exact dependency responsible: https://github.com/openSUSE/qem-dashboard/compare/13c6953b2c9626d1a13d78713ef2620657a390a1...e09992e423f83b0ef085108b60551ed811f882b2
Updated by kraih about 1 year ago
- Status changed from In Progress to Feedback
The main problem seems to be the latest playwright
update, but tap
also breaks prove
because it now uses TAP version 14, which is only supported in the latest bleeding edge CPAN release.
Update: Both dependencies are pinned for now.
Updated by livdywan about 1 year ago
Update: Both dependencies are pinned for now.
Would be great if you can find some kind of issue or ticket for it to block on. So we can eventually revert (even if it takes several months).
Updated by kraih about 1 year ago
livdywan wrote in #note-11:
Update: Both dependencies are pinned for now.
Would be great if you can find some kind of issue or ticket for it to block on. So we can eventually revert (even if it takes several months).
I've unpinned tap
at least, which can be made to work with a new version of prove
(Test::Harness 3.48). But i don't know the actual root cause for the Playwright issue, so can't block on a ticket there. Only way to know really is to try a test run every time there's a new release, and then unpin if it works.
Updated by kraih about 1 year ago
So, i've tracked the problem down to playwright 1.38.0, which seems to be incompatible with the current playwright GitHub Action. I expect this to resolve itself in time. https://github.com/openSUSE/qem-dashboard/actions/runs/6481566075/job/17599342276
The error already changed, so probably something about the browser build:
prove -l -v t/*.t.js
Error: Wide character in print at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/TAP/Formatter/Base.pm line 424.
t/ui.t.js ..
TAP version 14
# Subtest: Test dashboard ui
not ok 1 - browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium-1080/chrome-linux/chrome ╔═════════════════════════════════════════════════════════════════════════╗ ║ Looks like Playwright Test or Playwright was just installed or updated. ║ ║ Please run the following command to download new browsers: ║ ║ ║ ║ npx playwright install ║ ║ ║ ║ <3 Playwright Team ║ ╚═════════════════════════════════════════════════════════════════════════╝
No subtests run
Test Summary Report
-------------------
t/ui.t.js (Wstat: (none) Tests: 0 Failed: 0)
Error: Parse errors: Badly formed hash line: '' at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/TAP/Parser/YAMLish/Reader.pm line 244.
No plan found in TAP output
Files=1, Tests=0, 0 wallclock secs ( 0.02 usr + 0.01 sys = 0.03 CPU)
Result: FAIL
Error: Process completed with exit code 1.
For now i've bumped the pinned playwright version to 1.37.1, the last known good release.
Updated by kraih about 1 year ago
I completely missed that the playwright GitHub Action is deprecated. So it probably will not get fixed and we need to switch to the new CLI tools.
Updated by kraih about 1 year ago
- Status changed from In Progress to Resolved
And fixed by replacing the GitHub Action completely: https://github.com/openSUSE/qem-dashboard/commit/f61619163ee0583cb78146188c8acae685cc35e8