Project

General

Profile

Actions

action #137651

closed

[qem-dashboard] CI tests using Playwright are broken

Added by kraih 7 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2023-10-09
Due date:
% Done:

0%

Estimated time:

Description

Observation

While working on #132689 i noticed that the Playwright UI tests currently do not work properly and hang for 5 hours before failing with this error:

Run prove -l -v t/*.t.js
Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory
Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.
Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.
Error: The operation was canceled.

Locally the test passes, so this should be GitHub Action specific in some way. But this will block any 3rd party contributions for the time being.

Acceptance criteria

  • AC1: Tests pass in GitHub Action

Suggestions

  • Check if there are new best practices for using Playwright in GitHub Actions.
  • Check if there are any known IO::Socket::SSL errors that could cause this.
  • Maybe Try to SSH into a GitHub Action run.
Actions #1

Updated by tinita 7 months ago

The blib in the error message blib/lib/Net/SSLeay.pm is suspicious. Tests are run with prove -l, so blib can't come from there.

Actions #2

Updated by kraih 7 months ago

Also opened a ticket upstream for the deprecation warning, it's harmless in the context of this issue though. https://github.com/jhthorsen/mojolicious-plugin-webpack/issues/16

Actions #3

Updated by livdywan 7 months ago

kraih wrote in #note-2:

Also opened a ticket upstream for the deprecation warning, it's harmless in the context of this issue though. https://github.com/jhthorsen/mojolicious-plugin-webpack/issues/16

The logs claim that this is the fatal error, though? Supposedly the cover issue is being ignored?

Actions #4

Updated by kraih 7 months ago

livdywan wrote in #note-3:

The logs claim that this is the fatal error, though? Supposedly the cover issue is being ignored?

Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.

Actions #5

Updated by kraih 7 months ago

  • Status changed from New to In Progress
Actions #6

Updated by livdywan 7 months ago

Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.

Run prove -l -v t/*.t.js
Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory
Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.

Says ignored.

Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.

Says Error.

Error: The operation was canceled.
Actions #7

Updated by kraih 7 months ago

livdywan wrote in #note-6:

Where do the logs claim that? Please elaborate how you got to that conclusion, because i'm not seeing it.

Run prove -l -v t/*.t.js
Devel::Cover: Warning: can't open blib/lib/Net/SSLeay.pm for MD5 digest: No such file or directory
Devel::Cover: Can't find file "blib/lib/Net/SSLeay.pm" (blib/lib/Net/SSLeay.pm (autosplit into blib/lib/auto/Net/SSLeay/randomize.al)): ignored.

Says ignored.

Error: Mojo::File::spurt is deprecated in favor of Mojo::File::spew at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/Mojo/Alien/webpack.pm line 187.

Says Error.

Error: The operation was canceled.

I'm afraid you've misinterpreted the logs. Error only means that this has been printed to STDERR in the GitHub Action. Pass or failure of the step is decided by the final exit code. This step was cancelled by GitHub because it somehow ended up in an infinite loop. The deprecation is a Perl warning printed to STDERR, and completely harmless here.

Actions #8

Updated by kraih 7 months ago

I suspect that the real problem is actually something in one of the JavaScript dependencies that have recently been updated.

Actions #9

Updated by kraih 7 months ago

Downgrading the JavaScript dependencies resolved the problem, lets see if i can find the exact dependency responsible: https://github.com/openSUSE/qem-dashboard/compare/13c6953b2c9626d1a13d78713ef2620657a390a1...e09992e423f83b0ef085108b60551ed811f882b2

Actions #10

Updated by kraih 7 months ago

  • Status changed from In Progress to Feedback

The main problem seems to be the latest playwright update, but tap also breaks prove because it now uses TAP version 14, which is only supported in the latest bleeding edge CPAN release.

Update: Both dependencies are pinned for now.

Actions #11

Updated by livdywan 7 months ago

Update: Both dependencies are pinned for now.

Would be great if you can find some kind of issue or ticket for it to block on. So we can eventually revert (even if it takes several months).

Actions #12

Updated by kraih 7 months ago

livdywan wrote in #note-11:

Update: Both dependencies are pinned for now.

Would be great if you can find some kind of issue or ticket for it to block on. So we can eventually revert (even if it takes several months).

I've unpinned tap at least, which can be made to work with a new version of prove (Test::Harness 3.48). But i don't know the actual root cause for the Playwright issue, so can't block on a ticket there. Only way to know really is to try a test run every time there's a new release, and then unpin if it works.

Actions #13

Updated by kraih 7 months ago

So, i've tracked the problem down to playwright 1.38.0, which seems to be incompatible with the current playwright GitHub Action. I expect this to resolve itself in time. https://github.com/openSUSE/qem-dashboard/actions/runs/6481566075/job/17599342276

The error already changed, so probably something about the browser build:

prove -l -v t/*.t.js

Error: Wide character in print at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/TAP/Formatter/Base.pm line 424.
t/ui.t.js .. 
TAP version 14
# Subtest: Test dashboard ui
    not ok 1 - browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium-1080/chrome-linux/chrome ╔═════════════════════════════════════════════════════════════════════════╗ ║ Looks like Playwright Test or Playwright was just installed or updated. ║ ║ Please run the following command to download new browsers:              ║ ║                                                                         ║ ║     npx playwright install                                              ║ ║                                                                         ║ ║ <3 Playwright Team                                                      ║ ╚═════════════════════════════════════════════════════════════════════════╝
No subtests run 

Test Summary Report
-------------------
t/ui.t.js (Wstat: (none) Tests: 0 Failed: 0)
Error:   Parse errors: Badly formed hash line: '' at /opt/hostedtoolcache/perl/5.38.0/x64/lib/site_perl/5.38.0/TAP/Parser/YAMLish/Reader.pm line 244.

                No plan found in TAP output
Files=1, Tests=0,  0 wallclock secs ( 0.02 usr +  0.01 sys =  0.03 CPU)
Result: FAIL
Error: Process completed with exit code 1.

For now i've bumped the pinned playwright version to 1.37.1, the last known good release.

Actions #14

Updated by kraih 7 months ago

I completely missed that the playwright GitHub Action is deprecated. So it probably will not get fixed and we need to switch to the new CLI tools.

Actions #15

Updated by kraih 7 months ago

  • Status changed from Feedback to In Progress
Actions #16

Updated by kraih 7 months ago

  • Status changed from In Progress to Resolved
Actions #17

Updated by okurz 7 months ago

Nice!

Actions

Also available in: Atom PDF