Project

General

Profile

Actions

action #46679

closed

[functional][y] Reduce waiting timeout in yast2_hostnames while cursor is blinking

Added by oorlov about 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Target version:
SUSE QA - Milestone 23
Start date:
2019-01-25
Due date:
2019-03-12
% Done:

0%

Estimated time:
5.00 h
Difficulty:

Description

Currently, 'yast2_hostnames' test uses several 'wait_still_screen 1;', that actually wait for 30 seconds each time, because the cursor is blinking with interval which is faster than 1 sec, so that test assumes that screen is changing every time.

Find a proper solution to avoid that 30 sec waiting, but at the same time do not reduce the stability of the test.

Acceptance criteria

  1. Test doesn't wait when is not required

Suggestions:

  1. Remove these waits at all, but then ensure that the test is not started sporadically failing; OR
  2. Find proper similarity_level to not count cursor blinking as screen change, but text entering should still be counted as screen change; OR
  3. Find another from 'wait_still_screen' solution.

Related issues 1 (0 open1 closed)

Related to openQA Tests - action #48755: Replace `wait_screen_change` with `assert_screen_change` in a standard scenarioRejectedmgriessmeier2016-01-07

Actions
Actions #1

Updated by okurz about 5 years ago

  • Due date set to 2019-02-26
  • Status changed from New to Workable
  • Target version set to Milestone 23

Thank you for a good observation. Your suggestions are sound :)

Actions #2

Updated by mloviska about 5 years ago

  • Status changed from Workable to In Progress
  • Assignee set to mloviska
Actions #3

Updated by mloviska about 5 years ago

Why is the test suite soft failing? What we should expect in this screen?
https://openqa.opensuse.org/tests/852466#step/yast2_hostnames/10

Actions #4

Updated by mloviska about 5 years ago

I can see a bit different title of window, but the matched area is the same
https://openqa.opensuse.org/tests/493639#step/yast2_hostnames/17

Actions #5

Updated by mloviska about 5 years ago

Leap 15.1 does not soft fail and it seems to be quite the same design.
https://openqa.opensuse.org/tests/851374#step/yast2_hostnames/10

Actions #6

Updated by riafarov about 5 years ago

  • Description updated (diff)
  • Estimated time set to 5.00 h
Actions #7

Updated by mloviska about 5 years ago

Actions #8

Updated by mloviska about 5 years ago

  • Status changed from In Progress to Feedback
Actions #9

Updated by riafarov about 5 years ago

  • Due date changed from 2019-02-26 to 2019-03-12

Need to check when we get deployment to osd and merge test code changes.
As needles are merged, so either we need to revert that change or proceed with PR, as it might only introduce some sporadic issues on OSD, but won't fail as incomplete.

Actions #10

Updated by riafarov about 5 years ago

[2019-03-08T16:19:10.118 CET] [debug] ||| starting yast2_hostnames tests/yast2_gui/yast2_hostnames.pm
[2019-03-08T16:22:54.623 CET] [debug] ||| finished yast2_hostnames yast2_gui at 2019-03-08 15:22:54 (224 s)

[2019-03-01T19:56:00.432 CET] [debug] ||| finished yast2_hostnames yast2_gui at 2019-03-01 18:56:00 (204 s)
Seems we haven't improved running time =(. @mloviska, sould you please take a look? Please note, that we have TIMEOUT_SCALE=5 set there, but seems your change didn't help =(

Actions #11

Updated by mloviska about 5 years ago

Dates look quite old in respect that the PR was merge 5 days ago

Actions #12

Updated by mloviska about 5 years ago

[2019-03-07T11:30:02.716 CET] [debug] <<< testapi::wait_serial(regexp=qr/6e4e4-\d+-/, timeout=90)
[2019-03-07T11:30:04.799 CET] [debug] >>> testapi::wait_serial: (?^:6e4e4-\d+-): ok
[2019-03-07T11:30:04.853 CET] [debug] ||| finished yast2_hostnames yast2_gui at 2019-03-07 10:30:04 (72 s)
[2019-03-07T11:30:04.855 CET] [debug] ||| starting yast2_lang tests/yast2_gui/yast2_lang.pm

https://openqa.opensuse.org/tests/873108/file/autoinst-log.txt

Actions #13

Updated by mloviska about 5 years ago

Seems like the schedule has changed

[2019-03-08T10:52:01.763 CET] [debug] scheduling isosize tests/installation/isosize.pm
[2019-03-08T10:52:01.767 CET] [debug] scheduling bootloader tests/installation/bootloader.pm
[2019-03-08T10:52:01.772 CET] [debug] scheduling welcome tests/installation/welcome.pm
[2019-03-08T10:52:01.773 CET] [debug] scheduling online_repos tests/installation/online_repos.pm
[2019-03-08T10:52:01.773 CET] [debug] scheduling installation_mode tests/installation/installation_mode.pm
[2019-03-08T10:52:01.774 CET] [debug] scheduling logpackages tests/installation/logpackages.pm
[2019-03-08T10:52:01.775 CET] [debug] scheduling system_role tests/installation/system_role.pm
[2019-03-08T10:52:01.778 CET] [debug] scheduling partitioning tests/installation/partitioning.pm
[2019-03-08T10:52:01.778 CET] [debug] scheduling partitioning_togglehome tests/installation/partitioning_togglehome.pm
[2019-03-08T10:52:01.779 CET] [debug] scheduling partitioning_finish tests/installation/partitioning_finish.pm
[2019-03-08T10:52:01.779 CET] [debug] scheduling installer_timezone tests/installation/installer_timezone.pm
[2019-03-08T10:52:01.780 CET] [debug] scheduling user_settings tests/installation/user_settings.pm
[2019-03-08T10:52:01.781 CET] [debug] scheduling installation_overview tests/installation/installation_overview.pm
[2019-03-08T10:52:01.782 CET] [debug] scheduling disable_grub_timeout tests/installation/disable_grub_timeout.pm
[2019-03-08T10:52:01.782 CET] [debug] scheduling start_install tests/installation/start_install.pm
[2019-03-08T10:52:01.784 CET] [debug] scheduling await_install tests/installation/await_install.pm
[2019-03-08T10:52:01.784 CET] [debug] scheduling logs_from_installation_system tests/installation/logs_from_installation_system.pm
[2019-03-08T10:52:01.785 CET] [debug] scheduling reboot_after_installation tests/installation/reboot_after_installation.pm
[2019-03-08T10:52:01.786 CET] [debug] scheduling grub_test tests/installation/grub_test.pm
[2019-03-08T10:52:01.787 CET] [debug] scheduling first_boot tests/installation/first_boot.pm
[2019-03-08T10:52:01.788 CET] [debug] scheduling zypper_clear_repos tests/update/zypper_clear_repos.pm
[2019-03-08T10:52:01.789 CET] [debug] scheduling zypper_ar tests/console/zypper_ar.pm
[2019-03-08T10:52:01.790 CET] [debug] scheduling zypper_ref tests/console/zypper_ref.pm
[2019-03-08T10:52:01.790 CET] [debug] scheduling prepare_system_for_update_tests tests/update/prepare_system_for_update_tests.pm
[2019-03-08T10:52:01.817 CET] [debug] scheduling updates_packagekit_gpk tests/update/updates_packagekit_gpk.pm
[2019-03-08T10:52:01.818 CET] [debug] scheduling check_system_is_updated tests/update/check_system_is_updated.pm
[2019-03-08T10:52:01.819 CET] [debug] scheduling system_prepare tests/console/system_prepare.pm
[2019-03-08T10:52:01.819 CET] [debug] scheduling check_network tests/console/check_network.pm
[2019-03-08T10:52:01.820 CET] [debug] scheduling system_state tests/console/system_state.pm
[2019-03-08T10:52:01.820 CET] [debug] scheduling prepare_test_data tests/console/prepare_test_data.pm
[2019-03-08T10:52:01.821 CET] [debug] scheduling consoletest_setup tests/console/consoletest_setup.pm
[2019-03-08T10:52:01.822 CET] [debug] scheduling keymap_or_locale tests/locale/keymap_or_locale.pm
[2019-03-08T10:52:01.823 CET] [debug] scheduling force_scheduled_tasks tests/console/force_scheduled_tasks.pm
[2019-03-08T10:52:01.823 CET] [debug] scheduling textinfo tests/console/textinfo.pm
[2019-03-08T10:52:01.824 CET] [debug] scheduling hostname tests/console/hostname.pm
[2019-03-08T10:52:01.825 CET] [debug] scheduling xorg_vt tests/console/xorg_vt.pm
[2019-03-08T10:52:01.825 CET] [debug] scheduling zypper_lr tests/console/zypper_lr.pm
[2019-03-08T10:52:01.826 CET] [debug] scheduling ncurses tests/console/ncurses.pm
[2019-03-08T10:52:01.828 CET] [debug] scheduling yast2_lan tests/console/yast2_lan.pm
[2019-03-08T10:52:01.829 CET] [debug] scheduling curl_https tests/console/curl_https.pm
[2019-03-08T10:52:01.830 CET] [debug] scheduling salt tests/console/salt.pm
[2019-03-08T10:52:01.830 CET] [debug] scheduling glibc_sanity tests/console/glibc_sanity.pm
[2019-03-08T10:52:01.831 CET] [debug] scheduling zypper_in tests/console/zypper_in.pm
[2019-03-08T10:52:01.832 CET] [debug] scheduling yast2_i tests/console/yast2_i.pm
[2019-03-08T10:52:01.833 CET] [debug] scheduling yast2_bootloader tests/console/yast2_bootloader.pm
[2019-03-08T10:52:01.833 CET] [debug] scheduling vim tests/console/vim.pm
[2019-03-08T10:52:01.834 CET] [debug] scheduling firewall_enabled tests/console/firewall_enabled.pm
[2019-03-08T10:52:01.835 CET] [debug] scheduling sshd tests/console/sshd.pm
[2019-03-08T10:52:01.835 CET] [debug] scheduling ssh_cleanup tests/console/ssh_cleanup.pm
[2019-03-08T10:52:01.836 CET] [debug] scheduling sshfs tests/console/sshfs.pm
[2019-03-08T10:52:01.836 CET] [debug] scheduling mtab tests/console/mtab.pm
[2019-03-08T10:52:01.837 CET] [debug] scheduling consoletest_finish tests/console/consoletest_finish.pm
[2019-03-08T10:52:01.838 CET] [debug] scheduling user_gui_login tests/x11/user_gui_login.pm
[2019-03-08T10:52:01.838 CET] [debug] scheduling desktop_runner tests/x11/desktop_runner.pm
[2019-03-08T10:52:01.839 CET] [debug] scheduling xterm tests/x11/xterm.pm
[2019-03-08T10:52:01.839 CET] [debug] scheduling sshxterm tests/x11/sshxterm.pm
[2019-03-08T10:52:01.840 CET] [debug] scheduling gnome_control_center tests/x11/gnome_control_center.pm
[2019-03-08T10:52:01.840 CET] [debug] scheduling gnome_tweak_tool tests/x11/gnome_tweak_tool.pm
[2019-03-08T10:52:01.841 CET] [debug] scheduling gnome_terminal tests/x11/gnome_terminal.pm
[2019-03-08T10:52:01.841 CET] [debug] scheduling gedit tests/x11/gedit.pm
[2019-03-08T10:52:01.842 CET] [debug] scheduling firefox tests/x11/firefox.pm
[2019-03-08T10:52:01.843 CET] [debug] scheduling firefox_audio tests/x11/firefox_audio.pm
[2019-03-08T10:52:01.843 CET] [debug] scheduling thunderbird tests/x11/thunderbird.pm
[2019-03-08T10:52:01.844 CET] [debug] scheduling chromium tests/x11/chromium.pm
[2019-03-08T10:52:01.844 CET] [debug] scheduling ooffice tests/x11/ooffice.pm
[2019-03-08T10:52:01.845 CET] [debug] scheduling oomath tests/x11/oomath.pm
[2019-03-08T10:52:01.845 CET] [debug] scheduling oocalc tests/x11/oocalc.pm
[2019-03-08T10:52:01.846 CET] [debug] scheduling glxgears tests/x11/glxgears.pm
[2019-03-08T10:52:01.846 CET] [debug] scheduling nautilus tests/x11/nautilus.pm
[2019-03-08T10:52:01.847 CET] [debug] scheduling gnome_music tests/x11/gnome_music.pm
[2019-03-08T10:52:01.848 CET] [debug] scheduling evolution tests/x11/evolution.pm
[2019-03-08T10:52:01.848 CET] [debug] scheduling desktop_mainmenu tests/x11/desktop_mainmenu.pm
[2019-03-08T10:52:01.849 CET] [debug] scheduling inkscape tests/x11/inkscape.pm
[2019-03-08T10:52:01.849 CET] [debug] scheduling gimp tests/x11/gimp.pm
[2019-03-08T10:52:01.850 CET] [debug] scheduling hexchat tests/x11/hexchat.pm
[2019-03-08T10:52:01.851 CET] [debug] scheduling vlc tests/x11/vlc.pm
[2019-03-08T10:52:01.851 CET] [debug] scheduling reboot_gnome tests/x11/reboot_gnome.pm
[2019-03-08T10:52:01.852 CET] [debug] scheduling cleanup_before_shutdown tests/shutdown/cleanup_before_shutdown.pm
[2019-03-08T10:52:01.853 CET] [debug] scheduling shutdown tests/shutdown/shutdown.pm
Actions #14

Updated by mloviska about 5 years ago

Maybe we can improve wait_screen_change calls.

[2019-03-08T16:22:30.437 CET] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/yast2_gui/yast2_hostnames.pm:50 called testapi::type_string
[2019-03-08T16:22:30.437 CET] [debug] <<< testapi::type_string(string='download.opensuse.org', max_interval=13, wait_screen_changes=0, wait_still_screen='0.05', timeout=5, similarity_level=38)
[2019-03-08T16:22:30.437 CET] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/yast2_gui/yast2_hostnames.pm:50 called testapi::type_string
[2019-03-08T16:22:30.437 CET] [debug] <<< testapi::wait_screen_change(timeout=10, similarity_level=50)
[2019-03-08T16:22:36.781 CET] [debug] waiting for screen change: 0 34.3682315050034
[2019-03-08T16:22:36.782 CET] [debug] >>> testapi::wait_screen_change: screen change seen at 0
[2019-03-08T16:22:36.782 CET] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/yast2_gui/yast2_hostnames.pm:50 called testapi::type_string
[2019-03-08T16:22:36.782 CET] [debug] <<< testapi::wait_still_screen(similarity_level=38, stilltime='0.05', timeout=5)
[2019-03-08T16:22:37.287 CET] [debug] >>> testapi::wait_still_screen: detected same image for 0.05 seconds, last detected similarity is 1000000
[2019-03-08T16:22:37.288 CET] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/yast2_gui/yast2_hostnames.pm:51 called testapi::send_key
[2019-03-08T16:22:37.288 CET] [debug] <<< testapi::send_key(key='tab', do_wait=0)
[2019-03-08T16:22:37.493 CET] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/yast2_gui/yast2_hostnames.pm:52 called testapi::type_string
[2019-03-08T16:22:37.493 CET] [debug] <<< testapi::type_string(string='download-srv', max_interval=13, wait_screen_changes=0, wait_still_screen='0.05', timeout=5, similarity_level=38)
[2019-03-08T16:22:37.494 CET] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/yast2_gui/yast2_hostnames.pm:52 called testapi::type_string
[2019-03-08T16:22:37.494 CET] [debug] <<< testapi::wait_screen_change(timeout=10, similarity_level=50)
[2019-03-08T16:22:41.129 CET] [debug] waiting for screen change: 0 36.4430905762242
[2019-03-08T16:22:41.129 CET] [debug] >>> testapi::wait_screen_change: screen change seen at 0
[2019-03-08T16:22:41.129 CET] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/yast2_gui/yast2_hostnames.pm:52 called testapi::type_string
[2019-03-08T16:22:41.129 CET] [debug] <<< testapi::wait_still_screen(similarity_level=38, stilltime='0.05', timeout=5)
[2019-03-08T16:22:41.634 CET] [debug] >>> testapi::wait_still_screen: detected same image for 0.05 seconds, last detected similarity is 46.1690757571849
Actions #15

Updated by okurz about 5 years ago

  • Related to action #48755: Replace `wait_screen_change` with `assert_screen_change` in a standard scenario added
Actions #16

Updated by okurz about 5 years ago

You might be interested in #48755 which I see as related as this is also about "not ignoring return codes". wait_screen_change will definitely wait some minimum time but the maximum waiting time can happen when no screen change was seen during the whole timeout period.

Actions #17

Updated by riafarov about 5 years ago

  • Status changed from Feedback to Resolved
Actions #18

Updated by okurz about 5 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_gui
https://openqa.suse.de/tests/2506882

Actions

Also available in: Atom PDF