QA » openQA Project » openQA Tests

name=okurz_poo109737_chromium; end=1 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

Created job #2287933: opensuse-15.3-DVD-Updates-x86_64-Build20220409-1-extra_tests_misc@64bit -> https://openqa.opensuse.org/t2287933

But the base qcow image is already pruned from 15.3 Updates. That's annoying. I bumped from the default 5 GB in the job group https://openqa.opensuse.org/admin/job_templates/80 to 80GB. I retriggered the image creation job https://openqa.opensuse.org/tests/2287934 first.

Also I did

name=okurz_poo109737_chromium; end=1 dry_run=echo openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

to get a dry run of command execution. I removed the --skip-chained-deps and called openqa-clone-job manually. This triggered image creation parent jobs as well. Waiting for https://openqa.opensuse.org/tests/2287937

EDIT: Passed. The child job took 5m13s. So I can do name=okurz_poo109737_chromium; end=200 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

-> https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium

Should await results and gather fail ratio in current state of tests before trying to change something, e.g. type slower. In case the issue can not be reproduced at all then likely the system is busy in the original scenario due to background load, e.g. memory depleted.

EDIT: 1/100 failed in with mistyped URL (https://openqa.opensuse.org/tests/2287996#step/chromium/20; 1 job failing in a later step, counting as passed). Triggering a bigger set for better statistics with name=okurz_poo109737_chromium; start=201 end=500 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium

Result: 4/500 failed with lost characters so in this case even slightly below 1% fail ratio, but reproducible. I would now changed the enter_cmd call to enter_cmd ..., max_interval => 4;

Created new PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14698

and triggered

name=okurz_poo109737_chromium_pr14698; start=001 end=010 openqa-clone-set https://openqa.opensuse.org/tests/2287796 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/chromium_poo109737

-> https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium_pr14698

I checked one old and one new job and saw that the runtime for the test module chromium increased from 2m30s to 3m30s

To reduce the slowness I changed to only type the first three characters slowly and test that:

name=okurz_poo109737_chromium_pr14698_only_first_three_characters_slow; start=011 end=500 openqa-clone-set https://openqa.opensuse.org/tests/2294630 $name SCHEDULE=tests/boot/boot_to_desktop,tests/x11/chromium CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/chromium_poo109737

previous runs have aborted as they have been aborted. Retriggered based on a current image
https://openqa.opensuse.org/tests/overview?build=okurz_poo109737_chromium_pr14698_only_first_three_characters_slow

https://openqa.suse.de/tests/8816367#step/chromium/26
Mistyped https://upload.wikimedia.org/wikipedia/commons/d/d0/OpenSUSE_Logo.svg

Another failure on Leap15.3 https://openqa.opensuse.org/tests/2377093#step/chromium/22

I'll give it a go

Unfortunately, it is still failing

• https://openqa.suse.de/tests/8909126#step/chromium/19
• https://openqa.suse.de/tests/8909739#step/chromium/20
• https://openqa.suse.de/tests/8909565#step/chromium/20
• https://openqa.suse.de/tests/8909110#step/chromium/21
• https://openqa.suse.de/tests/8909107#step/chromium/20
• https://openqa.suse.de/tests/8909565#step/chromium/20

cdywan wrote:

okurz wrote:

Ok, from the recent test failures we see again different problems, some missing characters in between, even before the 10th character, some missing at the end, some duplication of characters. We see the following ideas:

1. Just start chromium with the correct address from the starter instead of typing in the address bar a) and terminate each instance of the browser and start again

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15031

I discovered in the meanwhile that chrome://version is ignored on the command-line and another option we weren't aware of:

As per the upstream Chromium issue --allow-pre-commit-input was added for the type of flakiness we're seeing. And maybe this allows typing to work without more hacks or delays.

• https://openqa.opensuse.org/tests/2422503#step/chromium/17
• https://openqa.opensuse.org/tests/2420962#step/chromium/20

Unfortunately it still fails sometimes. In both cases a search is opened instead of chrome://version which probably means we need the esc key hack afterall - I had removed it because I thought the pre-commit setting would render it irrelevant (no pun intended).
And maybe it's best to also assert the urlbar state... I'm starting to think it's pointless to simplify the test for something I can never reproduce outside of a full schedule run.

maybe simply start directly with the html5test, close, open again in whatever other url?. More over, while I get the thing that we're trying to use systems the way an user would do, why not simply go the automated test way?:

#!perl -w
use strict;
use Log::Log4perl qw(:easy);
use WWW::Mechanize::Chrome;
Log::Log4perl->easy_init($ERROR);

my $mech = WWW::Mechanize::Chrome->new(launch_exe => 'chromium-browser');

$mech->get( 'chrome://version' );
$mech->sleep( 10 );
$mech->get(qq(https://html5test.opensuse.org));
$mech->sleep( 10 );
$mech->get(qq(https://duckduckgo.org));
$mech->sleep( 10 );
$mech->sendkeys( string => "test\r" );

Also keep in mind that you could disable gpu acceleration (just in case?), also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)

szarate wrote:

[…] also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)

it might also just be many background processes from previous test modules. This could be found out by checking if the error can be reproduced in a test scenario which only boots to desktop and starts chromium

okurz wrote:

szarate wrote:

[…] also keep in mind that the memory might also be trolling you along with the machine's cpu config, so try 4CPU x 4GB for instance on 100 runs or so... because the scenario you're testing, is (at least for me, a bit unrealistic for web browsers, except in kiosk mode)

it might also just be many background processes from previous test modules. This could be found out by checking if the error can be reproduced in a test scenario which only boots to desktop and starts chromium

It can't. At least I have not seen any failures with that, which is why I got rid of the extra needle assertions.

cdywan wrote:

It can't. At least I have not seen any failures with that, which is why I got rid of the extra needle assertions.

this might just mean not enough jobs in the sample