action #59882

test fails in evolution_mail_imap to find certificate screen - potential regression from os-autoinst "slower typing"?

Added by okurz 5 months ago. Updated 4 months ago.

Status:ResolvedStart date:15/11/2019
Priority:HighDue date:
Assignee:okurz% Done:


Category:Bugs in existing tests
Target version:openQA Project - Current Sprint



openQA test in scenario sle-15-SP1-Desktop-DVD-Updates-x86_64-qam-regression-message@64bit fails in

Test suite description


Fails since (at least) Build 20191114-2

Expected result

Last good: 20191114-1 (or more recent)

Further details

Always latest result in this scenario: latest

Related issues

Related to openQA Tests - action #60050: [desktop][sporadically] test fails in evolution_meeting_i... New 19/11/2019
Copied to openQA Tests - action #60047: [desktop] Update maintainer in test modules, e.g. "evolut... New 19/11/2019


#1 Updated by okurz 5 months ago expects the previous screen "receiving email" to be confirmed properly. This is why a certificate window is expected. As if the previous screen was not properly confirmed (yet). I will clone the job to a local instance with old os-autoinst. http://lord.arch/tests/2669# shows up fine with os-autoinst a7874588 and os-autoinst-distri-opensuse e8439dd

Triggered again with recent os-autoinst:

sudo rm -rf /var/lib/openqa/pool/10/ulogs/ && openqa-clone-job --skip-chained-deps --skip-download --host http://lord.arch WORKER_CLASS=qemu_x86_64_sle MAKETESTSNAPSHOTS=1 EXCLUDE_MODULES=window_system,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin  && tail -n 200 -F /var/lib/openqa/pool/10/autoinst-log.txt

The good news is: http://lord.arch/tests/2672#step/evolution_mail_imap/17 is fine.

updated SLE needles checkout and checking again, skipping snapshots does not work:

sudo rm -rf /var/lib/openqa/pool/1
0/ulogs/ && openqa-clone-job --skip-chained-deps --skip-download --host http://lord.arch
/t3597505 WORKER_CLASS=qemu_x86_64_sle MAKETESTSNAPSHOTS=1 EXCLUDE_MODULES=window_system,evolution_smoke,evolut
ird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 && tail -n 200 -
F /var/lib/openqa/pool/10/autoinst-log.txt

http://lord.arch/tests/2675 is just fine with os-autoinst 8fe5e5bd1bd01e2adcd154adc94914d6316d2f22, current origin/master, os-autoinst-distri-opensuse 5e61debf413f11a37a92b944e53fb5e546b47c1b from 2019-08. Triggered new test on fc10cc06d , passed http://lord.arch/tests/2676

Triggered multiple tests:

sudo rm -rf /var/lib/openqa/pool/10/ulogs/ && for i in {001..020}; do openqa-clone-job --skip-chained-deps --skip-download --host http://lord.arch WORKER_CLASS=qemu_x86_64_sle MAKETESTSNAPSHOTS=1 EXCLUDE_MODULES=window_system,evolution_smoke,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_$i; done && tail -n 200 -F /var/lib/openqa/pool/10/autoinst-log.txt

-> http://lord.arch/tests/overview?version=15-SP1&distri=sle&build=20191115-1

if they are stable then the problem might be in the previous modules that I skip.

Triggering one on production:

openqa-clone-job --skip-chained-deps --skip-download --within-instance BUILD= _GROUP=0 EXCLUDE_MODULES=window_system,evolution_smoke,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_okurz

Created job #3599460: sle-15-SP1-Desktop-DVD-Updates-x86_64-Build20191115-1-qam-regression-message@64bit -> -> also stable

Trying with the modules in before evolution_mail_imap included:

openqa-clone-job --skip-chained-deps --skip-download --within-instance BUILD= _GROUP=0 EXCLUDE_MODULES=evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_okurz

Created job #3599466: sle-15-SP1-Desktop-DVD-Updates-x86_64-Build20191115-1-qam-regression-message@64bit -> -> passed

and triggered 20 more:

for i in {001..020} ; do openqa-clone-job --skip-chained-deps --skip-download --within-instance BUILD= _GROUP=0 EXCLUDE_MODULES=evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_with_smoke_poo59882_okurz_$i _GROUP=0 BUILD=poo59882; done


for comparison without smoke and window_system:

for i in {001..020} ; do openqa-clone-job --skip-chained-deps --skip-download --within-instance BUILD= _GROUP=0 EXCLUDE_MODULES=window_system,evolution_smoke,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_okurz_$i _GROUP=0 BUILD=poo59882:1 ; done


So far lord.arch fails in 1/7 runs but for a different reason than the original in http://lord.arch/tests/overview?version=15-SP1&distri=sle&build=20191115-1 and on osd we have 8/18 with 4 times the original and 4 times even before that

Looking at a "good" one (fails later than the original issue)
I can see a sequence of key that already looks weird:

[2019-11-15T15:43:23.775 CET] [debug] >>> testapi::_handle_found_needle: found mail_meeting_trust_ca-20180629, similarity 1.00 @ 53/156
[2019-11-15T15:43:23.775 CET] [debug] /var/lib/openqa/cache/ called x11test::setup_imap
[2019-11-15T15:43:23.776 CET] [debug] <<< testapi::send_key(key='alt-a', do_wait=0, wait_screen_change=0)
[2019-11-15T15:43:24.111 CET] [debug] <<< testapi::wait_screen_change(timeout=10, similarity_level=50)
[2019-11-15T15:43:24.112 CET] [debug] <<< testapi::send_key(key='alt-n', wait_screen_change=0, do_wait=0)
[2019-11-15T15:43:24.447 CET] [debug] <<< testapi::send_key(key='ret', do_wait=0, wait_screen_change=0)
[2019-11-15T15:43:25.220 CET] [debug] >>> testapi::wait_screen_change: screen change seen at 1
[2019-11-15T15:43:25.220 CET] [debug] <<< testapi::send_key(key='ret', do_wait=0, wait_screen_change=0)
[2019-11-15T15:43:25.572 CET] [debug] <<< testapi::assert_screen(mustmatch=[
], timeout=30)
[2019-11-15T15:43:25.675 CET] [debug] no match: 29.9s, best candidate: evolution_meeting_pop-evolution_wizard-receiving-opts-20160712 (0.00)

So there are multiple key presses which all seem to have no effect in the "good" case as can be seen in the video because after the screen "mail_meeting_trust_ca-20180629" and pressing "alt-a" to accept it the next one is simply "evolution_meeting_pop-evolution_wizard-receiving-opts-20160712", so the sequence "alt-n", "ret", "ret" does not have any effect here but I suspect it does have an effect in the "bad" case. And it could be more likely that the hotkeys are now actually received by the system due to making hotkeys more stable (by making them a bit "slower").

So I suggest to change the test in this regard to only send the keys that are necessary.

for i in {001..020} ; do openqa-clone-job --skip-chained-deps --skip-download --within-instance BUILD= _GROUP=0 EXCLUDE_MODULES=window_system,evolution_smoke,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_okurz_fix_73c71d7ad_$i _GROUP=0 BUILD=poo59882:2 CASEDIR= ; done


crosscheck on 12sp4: , on tumbleweed the scenario seems to be not cared about anyway :(

lord is more stable, probably due to lower performance than osd workers. created.

#2 Updated by okurz 5 months ago

  • Status changed from In Progress to Feedback

#3 Updated by okurz 4 months ago

  • Status changed from Feedback to Resolved

PR merged. Latest job also fails elsewhere. I talked to yfjiang and we agreed I can assign tickets to him to find new maintainers for the test modules and then try to stabilize the tests again after the behaviour of the backend changed.

#4 Updated by okurz 4 months ago

  • Copied to action #60047: [desktop] Update maintainer in test modules, e.g. "evolution_*" added

#5 Updated by okurz 4 months ago

  • Related to action #60050: [desktop][sporadically] test fails in evolution_meeting_imap and other test modules, potential changed behaviour due to recent "os-autoinst" change added

Also available in: Atom PDF