action #109737
[opensuse][sporadic] test fails in chromium due to lost characters when typing in the address bar size:M
0%
Description
Observation¶
openQA test in scenario opensuse-15.3-DVD-Updates-x86_64-gnome@64bit-2G fails in
chromium
due to lost characters when typing in the address bar which triggers an unexpected consent dialog shown for the page google.com instead of the expected "about" dialog.
The problem is triggered by the test API command
enter_cmd "chrome://version ";
which in some cases is not correctly typed. For example in https://openqa.opensuse.org/tests/2287780#step/chromium/19 it can be seen that a google search for "chrion" is conducted, likely because only some characters of "chrome://version " where typed, ending up with just the first three "chr" plus the last three (with our without the following space) "ion".
Acceptance criteria¶
- AC1: Chromium tests no longer fail sporadically
Reproducible¶
Fails sporadically. In https://openqa.opensuse.org/tests/2287780#next_previous I find a fail rate of roughly 3%.
Expected result¶
The test should ensure stable typing in the browser address bar or check for correct typing to prevent going to google at all or foresee the consent dialog. https://openqa.opensuse.org/tests/2287076#step/chromium/19 shows how the about dialog looks like when there is no consent dialog for google because in https://openqa.opensuse.org/tests/2287076#step/chromium/17 search.opensuse.org is shown.
Further details¶
Always latest result in this scenario: latest
Suggestions¶
- Type every single character very slowly
- Type first n(=10) characters slowly
- Switch off autocomplete suggestions in the URL
- If nothing works, just type all characters slowly again
Related issues
History
#2
Updated by okurz 3 months ago
name=okurz_poo109737_chromium; end=1 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium
Created job #2287933: opensuse-15.3-DVD-Updates-x86_64-Build20220409-1-extra_tests_misc@64bit -> https://openqa.opensuse.org/t2287933
But the base qcow image is already pruned from 15.3 Updates. That's annoying. I bumped from the default 5 GB in the job group https://openqa.opensuse.org/admin/job_templates/80 to 80GB. I retriggered the image creation job https://openqa.opensuse.org/tests/2287934 first.
Also I did
name=okurz_poo109737_chromium; end=1 dry_run=echo openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium
to get a dry run of command execution. I removed the --skip-chained-deps
and called openqa-clone-job manually. This triggered image creation parent jobs as well. Waiting for https://openqa.opensuse.org/tests/2287937
EDIT: Passed. The child job took 5m13s. So I can do name=okurz_poo109737_chromium; end=200 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium
-> https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium
Should await results and gather fail ratio in current state of tests before trying to change something, e.g. type slower. In case the issue can not be reproduced at all then likely the system is busy in the original scenario due to background load, e.g. memory depleted.
EDIT: 1/100 failed in with mistyped URL (https://openqa.opensuse.org/tests/2287996#step/chromium/20; 1 job failing in a later step, counting as passed). Triggering a bigger set for better statistics with name=okurz_poo109737_chromium; start=201 end=500 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium
#3
Updated by okurz 3 months ago
- Related to action #107632: [qe-core][leap][sporadic] Fix chromium test failing due to dropped keys added
#5
Updated by openqa_review 3 months ago
- Due date set to 2022-04-24
Setting due date based on mean cycle time of SUSE QE Tools
#6
Updated by okurz 3 months ago
Created new PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14698
and triggered
name=okurz_poo109737_chromium_pr14698; start=001 end=010 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/chromium_poo109737
-> https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium_pr14698
I checked one old and one new job and saw that the runtime for the test module chromium increased from 2m30s to 3m30s
#7
Updated by okurz 3 months ago
To reduce the slowness I changed to only type the first three characters slowly and test that:
name=okurz_poo109737_chromium_pr14698_only_first_three_characters_slow; start=011 end=500 openqa-clone-set https://openqa.opensuse.org/tests/2294630 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/chromium_poo109737
previous runs have aborted as they have been aborted. Retriggered based on a current image
https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium_pr14698_only_first_three_characters_slow
#8
Updated by okurz 3 months ago
- Status changed from In Progress to Feedback
All 500 jobs passed, PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14698 updated and ready for review.
#10
Updated by mloviska about 1 month ago
- Status changed from Resolved to Feedback
Seems like it still fails to type chrome://version
correctly https://openqa.suse.de/tests/8807706#step/chromium/19. Re-opening
#11
Updated by mloviska about 1 month ago
https://openqa.suse.de/tests/8816367#step/chromium/26
Mistyped https://upload.wikimedia.org/wikipedia/commons/d/d0/OpenSUSE_Logo.svg
#12
Updated by mloviska about 1 month ago
Another failure on Leap15.3 https://openqa.opensuse.org/tests/2377093#step/chromium/22
#13
Updated by okurz about 1 month ago
- Status changed from Feedback to New
- Assignee deleted (
okurz)
to be checked again
#14
Updated by cdywan about 1 month ago
- Subject changed from [opensuse][sporadic] test fails in chromium due to lost characters when typing in the address bar to [opensuse][sporadic] test fails in chromium due to lost characters when typing in the address bar size:M
- Description updated (diff)
- Status changed from New to Workable
#15
Updated by cdywan about 1 month ago
I'll give it a go
#16
Updated by cdywan about 1 month ago
- Status changed from Workable to In Progress
- Assignee set to cdywan
#17
Updated by openqa_review about 1 month ago
- Due date set to 2022-06-16
Setting due date based on mean cycle time of SUSE QE Tools
#18
Updated by okurz 25 days ago
- Status changed from In Progress to Feedback
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15004 merged. Please await verification from production, e.g. one night on o3 with no regressions observed.
#19
Updated by mloviska 24 days ago
Unfortunately, it is still failing
- https://openqa.suse.de/tests/8909126#step/chromium/19
- https://openqa.suse.de/tests/8909739#step/chromium/20
- https://openqa.suse.de/tests/8909565#step/chromium/20
- https://openqa.suse.de/tests/8909110#step/chromium/21
- https://openqa.suse.de/tests/8909107#step/chromium/20
- https://openqa.suse.de/tests/8909565#step/chromium/20
#20
Updated by okurz 24 days ago
- Status changed from Feedback to Workable
Ok, from the recent test failures we see again different problems, some missing characters in between, even before the 10th character, some missing at the end, some duplication of characters. We see the following ideas:
- Just start chromium with the correct address from the starter instead of typing in the address bar a) and terminate each instance of the browser and start again b) or just simplify the test again to only test a single URL
- instead of typing into the address bar use "xsel -i -p -b" to to copy-paste
- Try to find a way to disable auto-completion, e.g. similar as was done for Firefox in https://github.com/os-autoinst/os-autoinst-distri-opensuse/commit/def248c711503f2c6ed1276d61fce4c01b50eb6b
To trigger verification runs I suggest to find a test scenario that boots from a qcow file and then only trigger the relevant test module, e.g.
openqa-clone-custom-git-refspec https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15004 https://openqa.opensuse.org/tests/2405541 SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium
#21
Updated by cdywan 24 days ago
- Status changed from Workable to In Progress
okurz wrote:
Ok, from the recent test failures we see again different problems, some missing characters in between, even before the 10th character, some missing at the end, some duplication of characters. We see the following ideas:
- Just start chromium with the correct address from the starter instead of typing in the address bar a) and terminate each instance of the browser and start again
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15031
#22
Updated by cdywan 23 days ago
cdywan wrote:
okurz wrote:
Ok, from the recent test failures we see again different problems, some missing characters in between, even before the 10th character, some missing at the end, some duplication of characters. We see the following ideas:
- Just start chromium with the correct address from the starter instead of typing in the address bar a) and terminate each instance of the browser and start again
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15031
I discovered in the meanwhile that chrome://version is ignored on the command-line and another option we weren't aware of:
As per the upstream Chromium issue --allow-pre-commit-input
was added for the type of flakiness we're seeing. And maybe this allows typing to work without more hacks or delays.
#23
Updated by okurz 17 days ago
- Due date changed from 2022-06-16 to 2022-06-24
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15031 merged. Should be monitored over the next days.
#24
Updated by cdywan 12 days ago
- https://openqa.opensuse.org/tests/2422503#step/chromium/17
- https://openqa.opensuse.org/tests/2420962#step/chromium/20
Unfortunately it still fails sometimes. In both cases a search is opened instead of chrome://version
which probably means we need the esc key hack afterall - I had removed it because I thought the pre-commit setting would render it irrelevant (no pun intended).
And maybe it's best to also assert the urlbar state... I'm starting to think it's pointless to simplify the test for something I can never reproduce outside of a full schedule run.
#25
Updated by szarate 12 days ago
maybe simply start directly with the html5test, close, open again in whatever other url?. More over, while I get the thing that we're trying to use systems the way an user would do, why not simply go the automated test way?:
#!perl -w use strict; use Log::Log4perl qw(:easy); use WWW::Mechanize::Chrome; Log::Log4perl->easy_init($ERROR); my $mech = WWW::Mechanize::Chrome->new(launch_exe => 'chromium-browser'); $mech->get( 'chrome://version' ); $mech->sleep( 10 ); $mech->get(qq(https://html5test.opensuse.org)); $mech->sleep( 10 ); $mech->get(qq(https://duckduckgo.org)); $mech->sleep( 10 ); $mech->sendkeys( string => "test\r" );
Also keep in mind that you could disable gpu acceleration (just in case?), also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)
#26
Updated by okurz 11 days ago
szarate wrote:
[…] also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)
it might also just be many background processes from previous test modules. This could be found out by checking if the error can be reproduced in a test scenario which only boots to desktop and starts chromium
#27
Updated by cdywan 11 days ago
okurz wrote:
szarate wrote:
[…] also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)
it might also just be many background processes from previous test modules. This could be found out by checking if the error can be reproduced in a test scenario which only boots to desktop and starts chromium
It can't. At least I have not seen any failures with that, which is why I got rid of the extra needle assertions.
#29
Updated by cdywan 8 days ago
- Status changed from In Progress to Feedback
cdywan wrote:
- https://openqa.opensuse.org/tests/2422503#step/chromium/17
- https://openqa.opensuse.org/tests/2420962#step/chromium/20
Unfortunately it still fails sometimes. In both cases a search is opened instead of
chrome://version
which probably means we need the esc key hack afterall - I had removed it because I thought the pre-commit setting would render it irrelevant (no pun intended).
Monitoring jobs again with the above changes merged