Project

General

Profile

Actions
Copy link

action #164239

open

full-stack.t failing with "executeScript: unknown error: session deleted because of page crash at t/lib/OpenQA/SeleniumTest.pm:78"

Added by okurz 7 days ago. Updated 1 day ago.

Status:
Blocked
Priority:
Low
Assignee:
tinita
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2024-07-20
Due date:
% Done:

0%

Estimated time:
Tags:
CI, test, full-stack

Description

Observation

Observed in openQA CI tests both in PRs and master CI tests, e.g.
https://github.com/os-autoinst/openQA/pull/5779#issuecomment-2239102371
and in os-autoinst tests https://github.com/os-autoinst/os-autoinst/pull/2515#issuecomment-2251044169

[13:01:07] t/full-stack.t ......... Prototype mismatch: sub main::any: none vs (&@) at /usr/lib/perl5/5.26.1/Exporter.pm line 66.
 at t/full-stack.t line 25.
[13:01:07] t/full-stack.t ......... 36/? executeScript: unknown error: session deleted because of page crash at /home/squamata/project/t/lib/OpenQA/SeleniumTest.pm:78 at /home/squamata/project/t/lib/OpenQA/SeleniumTest.pm line 81.
        OpenQA::SeleniumTest::__ANON__(Test::Selenium::Chrome=HASH(0x5585d2665020), "Error while executing command: executeScript: unknown error: "..., HASH(0x5585d2f907c0), HASH(0x5585d3041450)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 356
        Selenium::Remote::Driver::catch {...} ("Error while executing command: executeScript: unknown error: "...) called at /usr/lib/perl5/vendor_perl/5.26.1/Try/Tiny.pm line 123
        Try::Tiny::try(CODE(0x5585d3164910), Try::Tiny::Catch=REF(0x5585d3164970)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 361
        Selenium::Remote::Driver::__ANON__(CODE(0x5585d0ffd220), Test::Selenium::Chrome=HASH(0x5585d2665020), HASH(0x5585d2f907c0), HASH(0x5585d3041450)) called at (eval 1557)[/usr/lib/perl5/vendor_perl/5.26.1/Class/Method/Modifiers.pm:89] line 1
        Selenium::Remote::Driver::__ANON__(Test::Selenium::Chrome=HASH(0x5585d2665020), HASH(0x5585d2f907c0), HASH(0x5585d3041450)) called at (eval 1559)[/usr/lib/perl5/vendor_perl/5.26.1/Class/Method/Modifiers.pm:148] line 2
        Selenium::Remote::Driver::_execute_command(Test::Selenium::Chrome=HASH(0x5585d2665020), HASH(0x5585d2f907c0), HASH(0x5585d3041450)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 1053
        Selenium::Remote::Driver::execute_script(Test::Selenium::Chrome=HASH(0x5585d2665020), "return window.jQuery \x{26}\x{26} jQuery.active === 0") called at /home/squamata/project/t/lib/OpenQA/SeleniumTest.pm line 147
        OpenQA::SeleniumTest::wait_for_ajax("msg", "wait_for_result_panel: waiting for '(?^u:State: running)' (co"...) called at /home/squamata/project/t/lib/OpenQA/Test/FullstackUtils.pm line 87
        OpenQA::Test::FullstackUtils::wait_for_result_panel(Test::Selenium::Chrome=HASH(0x5585d2665020), qr(State: running)u, 1) called at /home/squamata/project/t/lib/OpenQA/Test/FullstackUtils.pm line 108
        OpenQA::Test::FullstackUtils::wait_for_job_running(Test::Selenium::Chrome=HASH(0x5585d2665020), 1) called at t/full-stack.t line 338
        main::__ANON__() called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309
        eval {...} called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309
        Test::Builder::subtest(Test::Builder=HASH(0x5585c65205e0), "Cache tests", CODE(0x5585d3040e38)) called at /usr/lib/perl5/5.26.1/Test/More.pm line 807
        Test::More::subtest("Cache tests", CODE(0x5585d3040e38)) called at t/full-stack.t line 448
[13:01:07] t/full-stack.t ......... 37/? # Tests were run but no plan was declared and done_testing() was not seen.

First bad from CI tests in master from 1d ago:
https://app.circleci.com/pipelines/github/os-autoinst/openQA/14053/workflows/01e1dbe5-af68-49f4-8a3e-ad24fb39bf42/jobs/132357
with packages installed https://circleci-tasks-prod.s3.us-east-1.amazonaws.com/storage/artifacts/3f1e85c4-c546-442b-88e5-4343d260c2f4/0ca21dbe-25ad-4f52-86dd-f1eed597e34a/0/artifacts/packages.txt

Expected result

Last good from CI tests in master from 2d ago:
https://app.circleci.com/pipelines/github/os-autoinst/openQA/14048/workflows/96f1def0-eac9-40db-ba85-60b39b930227/jobs/132314
with packages installed https://circleci-tasks-prod.s3.us-east-1.amazonaws.com/storage/artifacts/3f1e85c4-c546-442b-88e5-4343d260c2f4/ff48df68-9ff6-4b4c-9c0c-537e97e67103/0/artifacts/packages.txt
no difference to "first bad"

Rollback steps

Actions
Copy link
 #1

Updated by tinita 7 days ago · Edited

Could it be a new version of chromium/chromedriver?
I get a failure as well on the latest 15.6 image, but it looks different.
But it fails around the same test:

ok 36 - Cache tests                                                                                                                                                                                               
executeScript: Server returned error message read timeout at /usr/lib/perl5/vendor_perl/5.26.1/Net/HTTP/Methods.pm line 268. at repos/openQA/t/lib/OpenQA/SeleniumTest.pm:78 at repos/openQA/t/lib/OpenQA/SeleniumTest.pm line 81.                                                                                                                                              
        OpenQA::SeleniumTest::__ANON__(Test::Selenium::Chrome=HASH(0x55dca8e5c140), "Error while executing command: executeScript: Server returned"..., HASH(0x55dca9214370), HASH(0x55dca9206e90)) called at /usr
/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 356                                                                                                                                                  
        Selenium::Remote::Driver::catch {...} ("Error while executing command: executeScript: Server returned"...) called at /usr/lib/perl5/vendor_perl/5.26.1/Try/Tiny.pm line 123                               
        Try::Tiny::try(CODE(0x55dca92042c8), Try::Tiny::Catch=REF(0x55dca9214730)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm line 361                                                 
        Selenium::Remote::Driver::__ANON__(CODE(0x55dca75aa718), Test::Selenium::Chrome=HASH(0x55dca8e5c140), HASH(0x55dca9214370), HASH(0x55dca9206e90)) called at (eval 1549) line 1                            
        Selenium::Remote::Driver::__ANON__(Test::Selenium::Chrome=HASH(0x55dca8e5c140), HASH(0x55dca9214370), HASH(0x55dca9206e90)) called at (eval 1551) line 2                                                  
        Selenium::Remote::Driver::_execute_command(Test::Selenium::Chrome=HASH(0x55dca8e5c140), HASH(0x55dca9214370), HASH(0x55dca9206e90)) called at /usr/lib/perl5/vendor_perl/5.26.1/Selenium/Remote/Driver.pm 
line 1053                                                                                                
        Selenium::Remote::Driver::execute_script(Test::Selenium::Chrome=HASH(0x55dca8e5c140), "return window.jQuery \x{26}\x{26} jQuery.active === 0") called at /home/tina/openqa-toolbox/repos/openQA/t/lib/Open
QA/SeleniumTest.pm line 147                                                                                                                                                                                       
        OpenQA::SeleniumTest::wait_for_ajax("msg", "wait_for_result_panel: waiting for '(?^u:Result: passed)' (co"...) called at /home/tina/openqa-toolbox/repos/openQA/t/lib/OpenQA/Test/FullstackUtils.pm line 8
7                                                                                                        
        OpenQA::Test::FullstackUtils::wait_for_result_panel(Test::Selenium::Chrome=HASH(0x55dca8e5c140), qr(Result: passed)u, "job 5") called at t/full-stack.t line 351                                          
        main::__ANON__() called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309                                                                                                                                 
        eval {...} called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309                                                                                                                                       
        Test::Builder::subtest(Test::Builder=HASH(0x55dc9e4bd410), "results of test 5", CODE(0x55dca91f1700)) called at /usr/lib/perl5/5.26.1/Test/More.pm line 807                                               
        Test::More::subtest("results of test 5", CODE(0x55dca91f1700)) called at t/full-stack.t line 389                                                                                                          
        main::__ANON__() called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309                        
        eval {...} called at /usr/lib/perl5/5.26.1/Test/Builder.pm line 309                                                                                                                                       
        Test::Builder::subtest(Test::Builder=HASH(0x55dc9e4bd410), "Cache tests", CODE(0x55dca9109438)) called at /usr/lib/perl5/5.26.1/Test/More.pm line 807
        Test::More::subtest("Cache tests", CODE(0x55dca9109438)) called at t/full-stack.t line 448

I have:

chromium-126.0.6478.126-bp156.2.6.1.x86_64
chromedriver-126.0.6478.126-bp156.2.6.1.x86_64

That's the same as in the latest registry.opensuse.org/devel/openqa/ci/containers/base:latest

Actions
Copy link
 #2

Updated by tinita 7 days ago

To me it doesn't look sporadic, btw. Looking at https://app.circleci.com/pipelines/github/os-autoinst/openQA it started to fail consistently at a certain point.

Actions
Copy link
 #3

Updated by okurz 7 days ago

  • Tags changed from CI, sporadic, test, full-stack to CI, test, full-stack
  • Subject changed from [sporadic] full-stack.t failing with "executeScript: unknown error: session deleted because of page crash at t/lib/OpenQA/SeleniumTest.pm:78" to full-stack.t failing with "executeScript: unknown error: session deleted because of page crash at t/lib/OpenQA/SeleniumTest.pm:78"
  • Description updated (diff)

yes, seems to be consistent now.

Actions
Copy link
 #4

Updated by tinita 5 days ago · Edited

https://bugzilla.suse.com/show_bug.cgi?id=1228180

After update from 125.0.6422.141 to 126.0.6478.126 the Chromium browser doesn't reliably load pages anymore (e.g. GMail, Confluence), loads forever or randomly shows "SIGSEGV" on white background.

actually marked as duplicate of
https://bugzilla.suse.com/show_bug.cgi?id=1227739

After updating to TW 20240711, which includes chromium 126, the browser keeps dumping core, [...]

Actions
Copy link
 #5

Updated by tinita 5 days ago

  • Status changed from New to In Progress
  • Assignee set to tinita
Actions
Copy link
 #6

Updated by tinita 5 days ago

  • Description updated (diff)

https://github.com/os-autoinst/openQA/pull/5783 Downgrade chromedriver to avoid broken version

Actions
Copy link
 #7

Updated by tinita 5 days ago

  • Description updated (diff)
  • Status changed from In Progress to Blocked
Actions
Copy link
 #8

Updated by tinita 1 day ago

Also in os-autoinst for the fullstack test: https://github.com/os-autoinst/os-autoinst/pull/2517 Downgrade chromedriver to avoid broken version

Actions
Copy link
 #9

Updated by tinita 1 day ago

  • Description updated (diff)
Actions
Copy link
 #10

Updated by okurz 1 day ago

  • Priority changed from High to Low

With the workaround applied we can lower prio

Actions
Copy link

Also available in: Atom PDF