Project

General

Profile

Actions

action #164239

closed

full-stack.t failing with "executeScript: unknown error: session deleted because of page crash at t/lib/OpenQA/SeleniumTest.pm:78"

Added by okurz 3 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2024-07-20
Due date:
% Done:

0%

Estimated time:

Description

Observation

Observed in openQA CI tests both in PRs and master CI tests, e.g.
https://github.com/os-autoinst/openQA/pull/5779#issuecomment-2239102371
and in os-autoinst tests https://github.com/os-autoinst/os-autoinst/pull/2515#issuecomment-2251044169

[13:01:07] t/full-stack.t ......... Prototype mismatch: sub main::any: none vs (&@) at /usr/lib/perl5/5.26.1/Exporter.pm line 66.
 at t/full-stack.t line 25.
[13:01:07] t/full-stack.t ......... 36/? executeScript: unknown error: session deleted because of page crash at /home/squamata/project/t/lib/OpenQA/SeleniumTest.pm:78 at /home/squamata/project/t/lib/OpenQA/SeleniumTest.pm line 81.
        OpenQA::SeleniumTest::__ANON__(Test::Selenium::Chrome=HASH(0x5585d2665020), "Error while executing command: executeScript: unknown error: "..., HASH(0x5585d2f907c0), HASH(0x5585d3041450)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 356
        Selenium::Remote::Driver::catch {...} ("Error while executing command: executeScript: unknown error: "...) called at /usr/lib/perl5/vendor_perl/5.26.1/Try/Tiny.pm line 123
        Try::Tiny::try(CODE(0x5585d3164910), Try::Tiny::Catch=REF(0x5585d3164970)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 361
        Selenium::Remote::Driver::__ANON__(CODE(0x5585d0ffd220), Test::Selenium::Chrome=HASH(0x5585d2665020), HASH(0x5585d2f907c0), HASH(0x5585d3041450)) called at (eval 1557)[/usr/lib/perl5/vendor_perl/5.26.1/Class/Method/Modifiers.pm:89] line 1
        Selenium::Remote::Driver::__ANON__(Test::Selenium::Chrome=HASH(0x5585d2665020), HASH(0x5585d2f907c0), HASH(0x5585d3041450)) called at (eval 1559)[/usr/lib/perl5/vendor_perl/5.26.1/Class/Method/Modifiers.pm:148] line 2
        Selenium::Remote::Driver::_execute_command(Test::Selenium::Chrome=HASH(0x5585d2665020), HASH(0x5585d2f907c0), HASH(0x5585d3041450)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 1053
        Selenium::Remote::Driver::execute_script(Test::Selenium::Chrome=HASH(0x5585d2665020), "return window.jQuery \x{26}\x{26} jQuery.active === 0") called at /home/squamata/project/t/lib/OpenQA/SeleniumTest.pm line 147
        OpenQA::SeleniumTest::wait_for_ajax("msg", "wait_for_result_panel: waiting for '(?^u:State: running)' (co"...) called at /home/squamata/project/t/lib/OpenQA/Test/FullstackUtils.pm line 87
        OpenQA::Test::FullstackUtils::wait_for_result_panel(Test::Selenium::Chrome=HASH(0x5585d2665020), qr(State: running)u, 1) called at /home/squamata/project/t/lib/OpenQA/Test/FullstackUtils.pm line 108
        OpenQA::Test::FullstackUtils::wait_for_job_running(Test::Selenium::Chrome=HASH(0x5585d2665020), 1) called at t/full-stack.t line 338
        main::__ANON__() called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309
        eval {...} called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309
        Test::Builder::subtest(Test::Builder=HASH(0x5585c65205e0), "Cache tests", CODE(0x5585d3040e38)) called at /usr/lib/perl5/5.26.1/Test/More.pm line 807
        Test::More::subtest("Cache tests", CODE(0x5585d3040e38)) called at t/full-stack.t line 448
[13:01:07] t/full-stack.t ......... 37/? # Tests were run but no plan was declared and done_testing() was not seen.

First bad from CI tests in master from 1d ago:
https://app.circleci.com/pipelines/github/os-autoinst/openQA/14053/workflows/01e1dbe5-af68-49f4-8a3e-ad24fb39bf42/jobs/132357
with packages installed https://circleci-tasks-prod.s3.us-east-1.amazonaws.com/storage/artifacts/3f1e85c4-c546-442b-88e5-4343d260c2f4/0ca21dbe-25ad-4f52-86dd-f1eed597e34a/0/artifacts/packages.txt

Expected result

Last good from CI tests in master from 2d ago:
https://app.circleci.com/pipelines/github/os-autoinst/openQA/14048/workflows/96f1def0-eac9-40db-ba85-60b39b930227/jobs/132314
with packages installed https://circleci-tasks-prod.s3.us-east-1.amazonaws.com/storage/artifacts/3f1e85c4-c546-442b-88e5-4343d260c2f4/ff48df68-9ff6-4b4c-9c0c-537e97e67103/0/artifacts/packages.txt
no difference to "first bad"

Rollback steps


Related issues 1 (0 open1 closed)

Related to openQA Project - action #165764: [ci] t/full-stack.t failing since 2024-08-23Resolvedmkittler2024-08-232024-09-07

Actions
Actions #1

Updated by tinita 3 months ago · Edited

Could it be a new version of chromium/chromedriver?
I get a failure as well on the latest 15.6 image, but it looks different.
But it fails around the same test:

ok 36 - Cache tests                                                                                                                                                                                               
executeScript: Server returned error message read timeout at /usr/lib/perl5/vendor_perl/5.26.1/Net/HTTP/Methods.pm line 268. at repos/openQA/t/lib/OpenQA/SeleniumTest.pm:78 at repos/openQA/t/lib/OpenQA/SeleniumTest.pm line 81.                                                                                                                                              
        OpenQA::SeleniumTest::__ANON__(Test::Selenium::Chrome=HASH(0x55dca8e5c140), "Error while executing command: executeScript: Server returned"..., HASH(0x55dca9214370), HASH(0x55dca9206e90)) called at /usr
/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 356                                                                                                                                                  
        Selenium::Remote::Driver::catch {...} ("Error while executing command: executeScript: Server returned"...) called at /usr/lib/perl5/vendor_perl/5.26.1/Try/Tiny.pm line 123                               
        Try::Tiny::try(CODE(0x55dca92042c8), Try::Tiny::Catch=REF(0x55dca9214730)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 361                                                 
        Selenium::Remote::Driver::__ANON__(CODE(0x55dca75aa718), Test::Selenium::Chrome=HASH(0x55dca8e5c140), HASH(0x55dca9214370), HASH(0x55dca9206e90)) called at (eval 1549) line 1                            
        Selenium::Remote::Driver::__ANON__(Test::Selenium::Chrome=HASH(0x55dca8e5c140), HASH(0x55dca9214370), HASH(0x55dca9206e90)) called at (eval 1551) line 2                                                  
        Selenium::Remote::Driver::_execute_command(Test::Selenium::Chrome=HASH(0x55dca8e5c140), HASH(0x55dca9214370), HASH(0x55dca9206e90)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm 
line 1053                                                                                                
        Selenium::Remote::Driver::execute_script(Test::Selenium::Chrome=HASH(0x55dca8e5c140), "return window.jQuery \x{26}\x{26} jQuery.active === 0") called at /home/tina/openqa-toolbox/repos/openQA/t/lib/Open
QA/SeleniumTest.pm line 147                                                                                                                                                                                       
        OpenQA::SeleniumTest::wait_for_ajax("msg", "wait_for_result_panel: waiting for '(?^u:Result: passed)' (co"...) called at /home/tina/openqa-toolbox/repos/openQA/t/lib/OpenQA/Test/FullstackUtils.pm line 8
7                                                                                                        
        OpenQA::Test::FullstackUtils::wait_for_result_panel(Test::Selenium::Chrome=HASH(0x55dca8e5c140), qr(Result: passed)u, "job 5") called at t/full-stack.t line 351                                          
        main::__ANON__() called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309                                                                                                                                 
        eval {...} called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309                                                                                                                                       
        Test::Builder::subtest(Test::Builder=HASH(0x55dc9e4bd410), "results of test 5", CODE(0x55dca91f1700)) called at /usr/lib/perl5/5.26.1/Test/More.pm line 807                                               
        Test::More::subtest("results of test 5", CODE(0x55dca91f1700)) called at t/full-stack.t line 389                                                                                                          
        main::__ANON__() called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309                        
        eval {...} called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309                                                                                                                                       
        Test::Builder::subtest(Test::Builder=HASH(0x55dc9e4bd410), "Cache tests", CODE(0x55dca9109438)) called at /usr/lib/perl5/5.26.1/Test/More.pm line 807
        Test::More::subtest("Cache tests", CODE(0x55dca9109438)) called at t/full-stack.t line 448

I have:

chromium-126.0.6478.126-bp156.2.6.1.x86_64
chromedriver-126.0.6478.126-bp156.2.6.1.x86_64

That's the same as in the latest registry.opensuse.org/devel/openqa/ci/containers/base:latest

Actions #2

Updated by tinita 3 months ago

To me it doesn't look sporadic, btw. Looking at https://app.circleci.com/pipelines/github/os-autoinst/openQA it started to fail consistently at a certain point.

Actions #3

Updated by okurz 3 months ago

  • Tags changed from CI, sporadic, test, full-stack to CI, test, full-stack
  • Subject changed from [sporadic] full-stack.t failing with "executeScript: unknown error: session deleted because of page crash at t/lib/OpenQA/SeleniumTest.pm:78" to full-stack.t failing with "executeScript: unknown error: session deleted because of page crash at t/lib/OpenQA/SeleniumTest.pm:78"
  • Description updated (diff)

yes, seems to be consistent now.

Actions #4

Updated by tinita 3 months ago · Edited

https://bugzilla.suse.com/show_bug.cgi?id=1228180

After update from 125.0.6422.141 to 126.0.6478.126 the Chromium browser doesn't reliably load pages anymore (e.g. GMail, Confluence), loads forever or randomly shows "SIGSEGV" on white background.

actually marked as duplicate of
https://bugzilla.suse.com/show_bug.cgi?id=1227739

After updating to TW 20240711, which includes chromium 126, the browser keeps dumping core, [...]

Actions #5

Updated by tinita 3 months ago

  • Status changed from New to In Progress
  • Assignee set to tinita
Actions #6

Updated by tinita 3 months ago

  • Description updated (diff)

https://github.com/os-autoinst/openQA/pull/5783 Downgrade chromedriver to avoid broken version

Actions #7

Updated by tinita 3 months ago

  • Description updated (diff)
  • Status changed from In Progress to Blocked
Actions #8

Updated by tinita 3 months ago

Also in os-autoinst for the fullstack test: https://github.com/os-autoinst/os-autoinst/pull/2517 Downgrade chromedriver to avoid broken version

Actions #9

Updated by tinita 3 months ago

  • Description updated (diff)
Actions #10

Updated by okurz 3 months ago

  • Priority changed from High to Low

With the workaround applied we can lower prio

Actions #11

Updated by tinita about 2 months ago · Edited

  • Status changed from Blocked to Feedback

According to https://bugzilla.suse.com/show_bug.cgi?id=1227739 fixed in chromium 127

We already have passing pull requests with that version as we only install the old package if we have version 126.

Remove workaround for broken chromium:

Actions #12

Updated by tinita about 2 months ago

  • Status changed from Feedback to Resolved

both merged, resolving

Actions #13

Updated by tinita about 2 months ago

  • Related to action #165764: [ci] t/full-stack.t failing since 2024-08-23 added
Actions

Also available in: Atom PDF