action #137651
closed
[qem-dashboard] CI tests using Playwright are broken
Added by kraih about 1 year ago.
Updated about 1 year ago.
Description
Observation¶
While working on #132689 i noticed that the Playwright UI tests currently do not work properly and hang for 5 hours before failing with this error:
Run prove -l -v t/*.t.js
Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory
Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.
Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.
Error: The operation was canceled.
Locally the test passes, so this should be GitHub Action specific in some way. But this will block any 3rd party contributions for the time being.
Acceptance criteria¶
- AC1: Tests pass in GitHub Action
Suggestions¶
- Check if there are new best practices for using Playwright in GitHub Actions.
- Check if there are any known IO::Socket::SSL errors that could cause this.
- Maybe Try to SSH into a GitHub Action run.
The blib
in the error message blib/lib/Net/SSLeay.pm
is suspicious. Tests are run with prove -l
, so blib can't come from there.
livdywan wrote in #note-3:
The logs claim that this is the fatal error, though? Supposedly the cover issue is being ignored?
Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.
- Status changed from New to In Progress
Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.
Run prove -l -v t/*.t.js
Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory
Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.
Says ignored.
Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.
Says Error.
Error: The operation was canceled.
livdywan wrote in #note-6:
Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.
Run prove -l -v t/*.t.js
Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory
Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.
Says ignored.
Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.
Says Error.
Error: The operation was canceled.
I'm afraid you've misinterpreted the logs. Error only means that this has been printed to STDERR in the GitHub Action. Pass or failure of the step is decided by the final exit code. This step was cancelled by GitHub because it somehow ended up in an infinite loop. The deprecation is a Perl warning printed to STDERR, and completely harmless here.
I suspect that the real problem is actually something in one of the JavaScript dependencies that have recently been updated.
- Status changed from In Progress to Feedback
The main problem seems to be the latest playwright
update, but tap
also breaks prove
because it now uses TAP version 14, which is only supported in the latest bleeding edge CPAN release.
Update: Both dependencies are pinned for now.
Update: Both dependencies are pinned for now.
Would be great if you can find some kind of issue or ticket for it to block on. So we can eventually revert (even if it takes several months).
livdywan wrote in #note-11:
Update: Both dependencies are pinned for now.
Would be great if you can find some kind of issue or ticket for it to block on. So we can eventually revert (even if it takes several months).
I've unpinned tap
at least, which can be made to work with a new version of prove
(Test::Harness 3.48). But i don't know the actual root cause for the Playwright issue, so can't block on a ticket there. Only way to know really is to try a test run every time there's a new release, and then unpin if it works.
So, i've tracked the problem down to playwright 1.38.0, which seems to be incompatible with the current playwright GitHub Action. I expect this to resolve itself in time. https://github.com/openSUSE/qem-dashboard/actions/runs/6481566075/job/17599342276
The error already changed, so probably something about the browser build:
prove -l -v t/*.t.js
Error: Wide character in print at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/TAP/Formatter/Base.pm line 424.
t/ui.t.js ..
TAP version 14
# Subtest: Test dashboard ui
not ok 1 - browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium-1080/chrome-linux/chrome ╔═════════════════════════════════════════════════════════════════════════╗ ║ Looks like Playwright Test or Playwright was just installed or updated. ║ ║ Please run the following command to download new browsers: ║ ║ ║ ║ npx playwright install ║ ║ ║ ║ <3 Playwright Team ║ ╚═════════════════════════════════════════════════════════════════════════╝
No subtests run
Test Summary Report
-------------------
t/ui.t.js (Wstat: (none) Tests: 0 Failed: 0)
Error: Parse errors: Badly formed hash line: '' at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/TAP/Parser/YAMLish/Reader.pm line 244.
No plan found in TAP output
Files=1, Tests=0, 0 wallclock secs ( 0.02 usr + 0.01 sys = 0.03 CPU)
Result: FAIL
Error: Process completed with exit code 1.
For now i've bumped the pinned playwright version to 1.37.1, the last known good release.
I completely missed that the playwright GitHub Action is deprecated. So it probably will not get fixed and we need to switch to the new CLI tools.
- Status changed from Feedback to In Progress
- Status changed from In Progress to Resolved
Also available in: Atom
PDF