QA » openQA Project » openQA Tests

https://openqa.suse.de/tests/3597505#step/evolution_mail_imap/20 expects the previous screen "receiving email" to be confirmed properly. This is why a certificate window is expected. As if the previous screen was not properly confirmed (yet). I will clone the job to a local instance with old os-autoinst. http://lord.arch/tests/2669# shows up fine with os-autoinst a7874588 and os-autoinst-distri-opensuse e8439dd

Triggered again with recent os-autoinst:

sudo rm -rf /var/lib/openqa/pool/10/ulogs/ && openqa-clone-job --skip-chained-deps --skip-download --host http://lord.arch https://openqa.suse.de/t3597505 WORKER_CLASS=qemu_x86_64_sle MAKETESTSNAPSHOTS=1 EXCLUDE_MODULES=window_system,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin  && tail -n 200 -F /var/lib/openqa/pool/10/autoinst-log.txt

The good news is: http://lord.arch/tests/2672#step/evolution_mail_imap/17 is fine.

updated SLE needles checkout and checking again, skipping snapshots does not work:

sudo rm -rf /var/lib/openqa/pool/1
0/ulogs/ && openqa-clone-job --skip-chained-deps --skip-download --host http://lord.arch https://openqa.suse.de
/t3597505 WORKER_CLASS=qemu_x86_64_sle MAKETESTSNAPSHOTS=1 EXCLUDE_MODULES=window_system,evolution_smoke,evolut
ion_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderb
ird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 && tail -n 200 -
F /var/lib/openqa/pool/10/autoinst-log.txt

http://lord.arch/tests/2675 is just fine with os-autoinst 8fe5e5bd1bd01e2adcd154adc94914d6316d2f22, current origin/master, os-autoinst-distri-opensuse 5e61debf413f11a37a92b944e53fb5e546b47c1b from 2019-08. Triggered new test on fc10cc06d , passed http://lord.arch/tests/2676

Triggered multiple tests:

sudo rm -rf /var/lib/openqa/pool/10/ulogs/ && for i in {001..020}; do openqa-clone-job --skip-chained-deps --skip-download --host http://lord.arch https://openqa.suse.de/t3597505 WORKER_CLASS=qemu_x86_64_sle MAKETESTSNAPSHOTS=1 EXCLUDE_MODULES=window_system,evolution_smoke,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_$i; done && tail -n 200 -F /var/lib/openqa/pool/10/autoinst-log.txt

-> http://lord.arch/tests/overview?version=15-SP1&distri=sle&build=20191115-1

if they are stable then the problem might be in the previous modules that I skip.

Triggering one on production:

openqa-clone-job --skip-chained-deps --skip-download --within-instance https://openqa.suse.de/t3597505 BUILD= _GROUP=0 EXCLUDE_MODULES=window_system,evolution_smoke,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_okurz

Created job #3599460: sle-15-SP1-Desktop-DVD-Updates-x86_64-Build20191115-1-qam-regression-message@64bit -> https://openqa.suse.de/t3599460 -> also stable

Trying with the modules in before evolution_mail_imap included:

openqa-clone-job --skip-chained-deps --skip-download --within-instance https://openqa.suse.de/t3597505 BUILD= _GROUP=0 EXCLUDE_MODULES=evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_okurz

Created job #3599466: sle-15-SP1-Desktop-DVD-Updates-x86_64-Build20191115-1-qam-regression-message@64bit -> https://openqa.suse.de/t3599466 -> passed

and triggered 20 more:

for i in {001..020} ; do openqa-clone-job --skip-chained-deps --skip-download --within-instance https://openqa.suse.de/t3597505 BUILD= _GROUP=0 EXCLUDE_MODULES=evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_with_smoke_poo59882_okurz_$i _GROUP=0 BUILD=poo59882; done

-> https://openqa.suse.de/tests/overview?version=15-SP1&distri=sle&build=poo59882

for comparison without smoke and window_system:

for i in {001..020} ; do openqa-clone-job --skip-chained-deps --skip-download --within-instance https://openqa.suse.de/t3597505 BUILD= _GROUP=0 EXCLUDE_MODULES=window_system,evolution_smoke,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_okurz_$i _GROUP=0 BUILD=poo59882:1 ; done

-> https://openqa.suse.de/tests/overview?version=15-SP1&distri=sle&build=poo59882%3A1

So far lord.arch fails in 1/7 runs but for a different reason than the original in http://lord.arch/tests/overview?version=15-SP1&distri=sle&build=20191115-1 and on osd we have 8/18 with 4 times the original and 4 times even before that https://openqa.suse.de/tests/overview?version=15-SP1&distri=sle&build=poo59882:1

Looking at a "good" one (fails later than the original issue) https://openqa.suse.de/tests/3599531/file/autoinst-log.txt
I can see a sequence of key that already looks weird:

[2019-11-15T15:43:23.775 CET] [debug] >>> testapi::_handle_found_needle: found mail_meeting_trust_ca-20180629, similarity 1.00 @ 53/156
[2019-11-15T15:43:23.775 CET] [debug] /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/x11/evolution/evolution_mail_imap.pm:28 called x11test::setup_imap
[2019-11-15T15:43:23.776 CET] [debug] <<< testapi::send_key(key='alt-a', do_wait=0, wait_screen_change=0)
[2019-11-15T15:43:24.111 CET] [debug] <<< testapi::wait_screen_change(timeout=10, similarity_level=50)
[2019-11-15T15:43:24.112 CET] [debug] <<< testapi::send_key(key='alt-n', wait_screen_change=0, do_wait=0)
[2019-11-15T15:43:24.447 CET] [debug] <<< testapi::send_key(key='ret', do_wait=0, wait_screen_change=0)
[2019-11-15T15:43:25.220 CET] [debug] >>> testapi::wait_screen_change: screen change seen at 1
[2019-11-15T15:43:25.220 CET] [debug] <<< testapi::send_key(key='ret', do_wait=0, wait_screen_change=0)
[2019-11-15T15:43:25.572 CET] [debug] <<< testapi::assert_screen(mustmatch=[
  'evolution_wizard-receiving-opts',
  'evolution_wizard-receiving-not-focused'
], timeout=30)
[2019-11-15T15:43:25.675 CET] [debug] no match: 29.9s, best candidate: evolution_meeting_pop-evolution_wizard-receiving-opts-20160712 (0.00)

So there are multiple key presses which all seem to have no effect in the "good" case as can be seen in the video https://openqa.suse.de/tests/3599531/file/video.ogv because after the screen "mail_meeting_trust_ca-20180629" and pressing "alt-a" to accept it the next one is simply "evolution_meeting_pop-evolution_wizard-receiving-opts-20160712", so the sequence "alt-n", "ret", "ret" does not have any effect here but I suspect it does have an effect in the "bad" case. And it could be more likely that the hotkeys are now actually received by the system due to https://github.com/os-autoinst/os-autoinst/pull/1260 making hotkeys more stable (by making them a bit "slower").

So I suggest to change the test in this regard to only send the keys that are necessary.

for i in {001..020} ; do openqa-clone-job --skip-chained-deps --skip-download --within-instance https://openqa.suse.de/t3597505 BUILD= _GROUP=0 EXCLUDE_MODULES=window_system,evolution_smoke,evolution_mail_pop,evolution_timezone_setup,evolution_meeting_imap,evolution_meeting_pop,thunderbird_install,thunderbird_imap,thunderbird_pop,groupwise,prep_pidgin,pidgin_IRC,clean_pidgin _SKIP_POST_FAIL_HOOKS=1 TEST=evolution_poo59882_okurz_fix_73c71d7ad_$i _GROUP=0 BUILD=poo59882:2 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/evolution ; done

-> https://openqa.suse.de/tests/overview?distri=sle&build=poo59882%3A2&version=15-SP1

crosscheck on 12sp4: https://openqa.suse.de/tests/3599703 , on tumbleweed the scenario seems to be not cared about anyway :(

lord is more stable, probably due to lower performance than osd workers.

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8937 created.