Project

General

Profile

action #109737

[opensuse][sporadic] test fails in chromium due to lost characters when typing in the address bar size:M

Added by okurz 3 months ago. Updated 5 days ago.

Status:
Feedback
Priority:
High
Assignee:
Category:
Bugs in existing tests
Target version:
Start date:
2022-04-09
Due date:
2022-07-08
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario opensuse-15.3-DVD-Updates-x86_64-gnome@64bit-2G fails in
chromium
due to lost characters when typing in the address bar which triggers an unexpected consent dialog shown for the page google.com instead of the expected "about" dialog.

The problem is triggered by the test API command

enter_cmd "chrome://version ";

which in some cases is not correctly typed. For example in https://openqa.opensuse.org/tests/2287780#step/chromium/19 it can be seen that a google search for "chrion" is conducted, likely because only some characters of "chrome://version " where typed, ending up with just the first three "chr" plus the last three (with our without the following space) "ion".

Acceptance criteria

  • AC1: Chromium tests no longer fail sporadically

Reproducible

Fails sporadically. In https://openqa.opensuse.org/tests/2287780#next_previous I find a fail rate of roughly 3%.

Expected result

The test should ensure stable typing in the browser address bar or check for correct typing to prevent going to google at all or foresee the consent dialog. https://openqa.opensuse.org/tests/2287076#step/chromium/19 shows how the about dialog looks like when there is no consent dialog for google because in https://openqa.opensuse.org/tests/2287076#step/chromium/17 search.opensuse.org is shown.

Further details

Always latest result in this scenario: latest

Suggestions

  • Type every single character very slowly
  • Type first n(=10) characters slowly
  • Switch off autocomplete suggestions in the URL
  • If nothing works, just type all characters slowly again

Related issues

Related to openQA Tests - action #107632: [qe-core][leap][sporadic] Fix chromium test failing due to dropped keysRejected

History

#1 Updated by okurz 3 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz
  • Target version set to Ready

#2 Updated by okurz 3 months ago

name=okurz_poo109737_chromium; end=1 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

Created job #2287933: opensuse-15.3-DVD-Updates-x86_64-Build20220409-1-extra_tests_misc@64bit -> https://openqa.opensuse.org/t2287933

But the base qcow image is already pruned from 15.3 Updates. That's annoying. I bumped from the default 5 GB in the job group https://openqa.opensuse.org/admin/job_templates/80 to 80GB. I retriggered the image creation job https://openqa.opensuse.org/tests/2287934 first.

Also I did

name=okurz_poo109737_chromium; end=1 dry_run=echo openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

to get a dry run of command execution. I removed the --skip-chained-deps and called openqa-clone-job manually. This triggered image creation parent jobs as well. Waiting for https://openqa.opensuse.org/tests/2287937

EDIT: Passed. The child job took 5m13s. So I can do name=okurz_poo109737_chromium; end=200 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

-> https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium

Should await results and gather fail ratio in current state of tests before trying to change something, e.g. type slower. In case the issue can not be reproduced at all then likely the system is busy in the original scenario due to background load, e.g. memory depleted.

EDIT: 1/100 failed in with mistyped URL (https://openqa.opensuse.org/tests/2287996#step/chromium/20; 1 job failing in a later step, counting as passed). Triggering a bigger set for better statistics with name=okurz_poo109737_chromium; start=201 end=500 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

#3 Updated by okurz 3 months ago

  • Related to action #107632: [qe-core][leap][sporadic] Fix chromium test failing due to dropped keys added

#4 Updated by okurz 3 months ago

Result: 4/500 failed with lost characters so in this case even slightly below 1% fail ratio, but reproducible. I would now changed the enter_cmd call to enter_cmd ..., max_interval => 4;

#5 Updated by openqa_review 3 months ago

  • Due date set to 2022-04-24

Setting due date based on mean cycle time of SUSE QE Tools

#6 Updated by okurz 3 months ago

Created new PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14698

and triggered

name=okurz_poo109737_chromium_pr14698; start=001 end=010 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/chromium_poo109737

-> https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium_pr14698

I checked one old and one new job and saw that the runtime for the test module chromium increased from 2m30s to 3m30s

#7 Updated by okurz 3 months ago

To reduce the slowness I changed to only type the first three characters slowly and test that:

name=okurz_poo109737_chromium_pr14698_only_first_three_characters_slow; start=011 end=500 openqa-clone-set https://openqa.opensuse.org/tests/2294630 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/chromium_poo109737

previous runs have aborted as they have been aborted. Retriggered based on a current image
https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium_pr14698_only_first_three_characters_slow

#8 Updated by okurz 3 months ago

  • Status changed from In Progress to Feedback

All 500 jobs passed, PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14698 updated and ready for review.

#9 Updated by okurz 2 months ago

  • Due date deleted (2022-04-24)
  • Status changed from Feedback to Resolved

There were no further reviews, merging myself. I didn't do any needle changes and did thousand of verification jobs so considering this resolved.

#10 Updated by mloviska about 1 month ago

  • Status changed from Resolved to Feedback

Seems like it still fails to type chrome://version correctly https://openqa.suse.de/tests/8807706#step/chromium/19. Re-opening

#11 Updated by mloviska about 1 month ago

https://openqa.suse.de/tests/8816367#step/chromium/26
Mistyped https://upload.wikimedia.org/wikipedia/commons/d/d0/OpenSUSE_Logo.svg

#13 Updated by okurz about 1 month ago

  • Status changed from Feedback to New
  • Assignee deleted (okurz)

to be checked again

#14 Updated by cdywan about 1 month ago

  • Subject changed from [opensuse][sporadic] test fails in chromium due to lost characters when typing in the address bar to [opensuse][sporadic] test fails in chromium due to lost characters when typing in the address bar size:M
  • Description updated (diff)
  • Status changed from New to Workable

#15 Updated by cdywan about 1 month ago

I'll give it a go

#16 Updated by cdywan about 1 month ago

  • Status changed from Workable to In Progress
  • Assignee set to cdywan

#17 Updated by openqa_review about 1 month ago

  • Due date set to 2022-06-16

Setting due date based on mean cycle time of SUSE QE Tools

#18 Updated by okurz 25 days ago

  • Status changed from In Progress to Feedback

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15004 merged. Please await verification from production, e.g. one night on o3 with no regressions observed.

#20 Updated by okurz 24 days ago

  • Status changed from Feedback to Workable

Ok, from the recent test failures we see again different problems, some missing characters in between, even before the 10th character, some missing at the end, some duplication of characters. We see the following ideas:

  1. Just start chromium with the correct address from the starter instead of typing in the address bar a) and terminate each instance of the browser and start again b) or just simplify the test again to only test a single URL
  2. instead of typing into the address bar use "xsel -i -p -b" to to copy-paste
  3. Try to find a way to disable auto-completion, e.g. similar as was done for Firefox in https://github.com/os-autoinst/os-autoinst-distri-opensuse/commit/def248c711503f2c6ed1276d61fce4c01b50eb6b

To trigger verification runs I suggest to find a test scenario that boots from a qcow file and then only trigger the relevant test module, e.g.

openqa-clone-custom-git-refspec https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15004 https://openqa.opensuse.org/tests/2405541 SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

#21 Updated by cdywan 24 days ago

  • Status changed from Workable to In Progress

okurz wrote:

Ok, from the recent test failures we see again different problems, some missing characters in between, even before the 10th character, some missing at the end, some duplication of characters. We see the following ideas:

  1. Just start chromium with the correct address from the starter instead of typing in the address bar a) and terminate each instance of the browser and start again

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15031

#22 Updated by cdywan 23 days ago

cdywan wrote:

okurz wrote:

Ok, from the recent test failures we see again different problems, some missing characters in between, even before the 10th character, some missing at the end, some duplication of characters. We see the following ideas:

  1. Just start chromium with the correct address from the starter instead of typing in the address bar a) and terminate each instance of the browser and start again

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15031

I discovered in the meanwhile that chrome://version is ignored on the command-line and another option we weren't aware of:

As per the upstream Chromium issue --allow-pre-commit-input was added for the type of flakiness we're seeing. And maybe this allows typing to work without more hacks or delays.

#23 Updated by okurz 17 days ago

  • Due date changed from 2022-06-16 to 2022-06-24

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15031 merged. Should be monitored over the next days.

#24 Updated by cdywan 12 days ago

Unfortunately it still fails sometimes. In both cases a search is opened instead of chrome://version which probably means we need the esc key hack afterall - I had removed it because I thought the pre-commit setting would render it irrelevant (no pun intended).
And maybe it's best to also assert the urlbar state... I'm starting to think it's pointless to simplify the test for something I can never reproduce outside of a full schedule run.

#25 Updated by szarate 12 days ago

maybe simply start directly with the html5test, close, open again in whatever other url?. More over, while I get the thing that we're trying to use systems the way an user would do, why not simply go the automated test way?:

#!perl -w
use strict;
use Log::Log4perl qw(:easy);
use WWW::Mechanize::Chrome;
Log::Log4perl->easy_init($ERROR);

my $mech = WWW::Mechanize::Chrome->new(launch_exe => 'chromium-browser');

$mech->get( 'chrome://version' );
$mech->sleep( 10 );
$mech->get(qq(https://html5test.opensuse.org));
$mech->sleep( 10 );
$mech->get(qq(https://duckduckgo.org));
$mech->sleep( 10 );
$mech->sendkeys( string => "test\r" );

Also keep in mind that you could disable gpu acceleration (just in case?), also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)

#26 Updated by okurz 11 days ago

szarate wrote:

[…] also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)

it might also just be many background processes from previous test modules. This could be found out by checking if the error can be reproduced in a test scenario which only boots to desktop and starts chromium

#27 Updated by cdywan 11 days ago

okurz wrote:

szarate wrote:

[…] also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)

it might also just be many background processes from previous test modules. This could be found out by checking if the error can be reproduced in a test scenario which only boots to desktop and starts chromium

It can't. At least I have not seen any failures with that, which is why I got rid of the extra needle assertions.

#28 Updated by okurz 11 days ago

cdywan wrote:

It can't. At least I have not seen any failures with that, which is why I got rid of the extra needle assertions.

this might just mean not enough jobs in the sample

#29 Updated by cdywan 8 days ago

  • Status changed from In Progress to Feedback

cdywan wrote:

Unfortunately it still fails sometimes. In both cases a search is opened instead of chrome://version which probably means we need the esc key hack afterall - I had removed it because I thought the pre-commit setting would render it irrelevant (no pun intended).

Monitoring jobs again with the above changes merged

#30 Updated by cdywan 5 days ago

  • Due date changed from 2022-06-24 to 2022-07-08

Bumping the due date til after hackweek, as per the previous comment

Also available in: Atom PDF