Project

General

Profile

action #68284

Selenium tests fail with "sendKeysToElement: Server returned error message" on CircleCI

Added by mkittler over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2020-06-22
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Selenium tests fail on CircleCI. The full error message looks like this:

[09:02:09] t/ui/10-tests_overview.t ............... 9/?  
Bailout called.  Further testing stopped:  sendKeysToElement: Server returned error message Server closed connection without sending any data back at /usr/lib/perl5/vendor_perl/5.26.1/Net/HTTP/Methods.pm line 391. at /home/squamata/project/t/ui/../lib/OpenQA/SeleniumTest.pm:103
make[2]: *** [Makefile:168: test-unit-and-integration] Terminated
make[1]: *** [Makefile:163: test-with-database] Terminated
make: *** [Makefile:137: test-ui] Terminated
Too long with no output (exceeded 10m0s): context deadline exceeded
  • Bailout called. … SeleniumTest.pm:103 is printed soon after the test is started. Then it hangs until the "deadline is exceeded" an the rest of the output is printed.
  • It seems that all tests using $driver->send_keys are affected. It doesn't seem to be a general timeout/limit because when reordering tests so 10-tests_overview.t runs at the beginning the test fails immediately.
  • The problem can not be reproduced locally under Tumbleweed.

Related issues

Related to openQA Project - action #93453: openQA CI: Certain Selenium tests are failingResolved2021-06-032021-06-18

History

#1 Updated by mkittler over 1 year ago

I can reproduce the problem locally within the Docker container used on CircleCI. Running a very simple test with only $driver->find_element('body')->send_keys('k'); via prove is enough to reproduce inside the container.

It didn't help to update some of the most obvious dependencies like chromedriver, perl-Net-HTTP and perl-Selenium-Remote-Driver.

I've monitoring the communication with tcpdump. Interestingly the command sendKeysToElement never shows up in the traffic. The request to send the key looks like this on TW:

POST /wd/hub/session/86019f417440b563f52645d9414d1d2e/element/0.5701700981523048-1/value HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: application/json
Host: 127.0.0.1:46847
User-Agent: libwww-perl/6.45
Content-Type: application/json; charset=utf-8
Content-Length: 26

{"text":"k","value":["k"]}

And like this within the CI container:

POST /wd/hub/session/0a66417b2d9a4cf4263b1f27dd716dbf/element/0.1713929545807995-1/value HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: application/json
Host: 127.0.0.1:56909
User-Agent: libwww-perl/6.31
Content-Type: application/json; charset=utf-8
Content-Length: 26

{"value":["k"],"text":"k"}

The answer looks like this on TW:

HTTP/1.1 200 OK
Content-Length:72
Content-Type:application/json; charset=utf-8
Connection:close

{"sessionId":"86019f417440b563f52645d9414d1d2e","status":0,"value":null}

Within the CI container's dump I couldn't find any matching answer HTTP response or any further mentioning of the session ID.

Within the container it is a little bit difficult to match listening ports and process IDs. But from the non-containerized run it is clear that these HTTP requests goes directly from our Perl process to the chromedriver process (which apparently does not respond correctly to the request which seems to be made correctly from the Perl side). So it is likely the chromedriver binary (or any dependency or influencing environment differences on the "chromedriver side") which makes the difference.

By the way, only the package (but not upstream) version differs between TW and CI:

chromedriver version within TW: 83.0.4103.97-1.1
chromedriver version within CI: 83.0.4103.97-lp151.2.96.1

#2 Updated by mkittler over 1 year ago

By the way, that's my test case if anyone wants to reproduce:

use Test::Most;
use FindBin;
use lib "$FindBin::Bin/../lib";
use Test::Warnings ':report_warnings';
use OpenQA::Test::Case;
use OpenQA::SeleniumTest;

my $test_case   = OpenQA::Test::Case->new;
my $schema_name = OpenQA::Test::Database->generate_schema_name;
my $schema      = $test_case->init_data(schema_name => $schema_name);
my $driver      = call_driver(sub {});

$driver->title_is('openQA', 'on main page');
$driver->find_element('body')->send_keys('k');

kill_driver();
done_testing();

#3 Updated by mkittler over 1 year ago

Downgrading chromedriver helps: zypper in --oldpackage chromedriver-81.0.4044.138-lp151.2.88.1 chromium-81.0.4044.138-lp151.2.88.1

As already mentioned: chromedriver 83.0.4103.97 works under TW. So likely the problem only occurs combined with an (older) version of some dependency.

For now I'll try to downgrade the package in the Dockerfile used to make the CI image. It is likely also worth filing a bug that the chromedriver provided by Leap is broken although it seems to be just a corner case which is broken.

#4 Updated by mkittler over 1 year ago

PR: https://github.com/os-autoinst/openQA/pull/3198

There seem to be even more severe problems with that Chromium version under Leap 15.1 so I've added a comment under the related ticket: https://bugzilla.opensuse.org/show_bug.cgi?id=1173107#c29
If the chromedriver problem is not related after all I can still create a separate ticket.

#5 Updated by mkittler over 1 year ago

Installing an old package version within the Dockerfile is not that simple: https://github.com/os-autoinst/openQA/pull/3200

Considering that a fixed package is already on the way I'd say we just wait for it.

#6 Updated by okurz over 1 year ago

The question is what's the ETA. force-installing the different version outside the container sounds like a viable alternative for now. https://build.opensuse.org/request/show/816801 is the SR for Leap 15.1. It was accepted already but so far podman run --rm -it registry.opensuse.org/opensuse/leap:15.1 /bin/sh -c 'zypper se --details chromium' yields 83.0.4103.106-lp151.2.101.1, the old version. I assume that a fix would only be published the soonest after 7 days in the "maintenance-test" repo. And https://build.opensuse.org/request/show/816975 is the according maintenance release request, still in review. And that references failed openQA tests in the chromium test module due to subtle font changes: https://openqa.opensuse.org/tests/1315699#step/chromium/21

So

  1. maintenance would probably benefit from our help
  2. it will take longer

#7 Updated by mkittler over 1 year ago

PR for manually downgrading before running tests: https://github.com/os-autoinst/openQA/pull/3207

It also looks like they've already created a new needle: https://github.com/os-autoinst/os-autoinst-needles-opensuse/commit/275aee9559f353b8a2b907cf260ff524b6d5ae85
So let's see whether https://openqa.opensuse.org/tests/1317062 will pass.

#8 Updated by cdywan about 1 year ago

  • Status changed from In Progress to Feedback

The work-around seem to do the trick.

#9 Updated by okurz about 1 year ago

  • Category set to Concrete Bugs

#10 Updated by mkittler about 1 year ago

I can now reproduce the issue under Tumbleweed. At least it looks very similar:

    ok 17 - 4 input fields
Bailout called.  Further testing stopped:  sendKeysToElement: Server returned error message Server closed connection without sending any data back at /usr/lib/perl5/vendor_perl/5.30.1/Net/HTTP/Methods.pm line 391. at /hdd/openqa-devel/repos/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:103
Bail out!  sendKeysToElement: Server returned error message Server closed connection without sending any data back at /usr/lib/perl5/vendor_perl/5.30.1/Net/HTTP/Methods.pm line 391. at /hdd/openqa-devel/repos/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:103

When trying to run it in headless mode it even fails to do anything:

Could not obtain new session: unknown error: Chrome failed to start: exited abnormally.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/lib64/chromium/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
  (Driver info: chromedriver=84.0.4147.89 (19abfe7bcba9318a0b2a6bc6634a67fc834aa592-refs/branch-heads/4147@{#852}),platform=Linux 5.7.9-1-default x86_64) at /hdd/openqa-devel/repos/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm line 119.
No subtests run

I'm using chromedriver 84.0.4147.89-1.1 as provided by TW.

#11 Updated by mkittler about 1 year ago

I've created a TW bug which also include a workaround: https://bugzilla.opensuse.org/show_bug.cgi?id=1174723

#12 Updated by okurz about 1 year ago

right, we still have this ticket :) And I agree that it is a good idea to reuse that.

#13 Updated by mkittler about 1 year ago

#14 Updated by mkittler about 1 year ago

  • Status changed from Feedback to Resolved

Tests of the reverted workaround passed and I've merged it.

#15 Updated by tinita 4 months ago

  • Related to action #93453: openQA CI: Certain Selenium tests are failing added

Also available in: Atom PDF