action #34471

[functional][opensuse][u][medium] too early matching in too generic needle text-login-20160812

Added by okurz almost 2 years ago. Updated 24 days ago.

Status:NewStart date:08/04/2018
Priority:NormalDue date:
Assignee:-% Done:

80%

Category:Enhancement to existing testsEstimated time:42.00 hours
Target version:SUSE QA tests - Milestone 30
Difficulty:medium
Duration:

Description

Observation

openQA test in scenario opensuse-42.3-Updates-x86_64-textmode@64bit fails in
zypper_up

[7 Apr 2018 21:01:23] <tacit> okurz: Somewhat more frequent unstable failure: https://openqa.opensuse.org/tests/652064#step/zypper_up/4    suspecting timing here as the root username is entered before the login promot
[7 Apr 2018 21:57:16] <okurz> tacit: well, problem is the not so good needle in https://openqa.opensuse.org/tests/652064#step/zypper_up/1 . as you can see on the '(i)' icon the test looks for tty6-selected and this is what we should use. What happens now that the test expects tt6, checks for just the generic needle, is happy with tty1, then the VM actually switches to tty6 but in the process the root name is already typed, only after that the getty is 
[7 Apr 2018 21:57:16] <okurz> activated. But you know what, I think I have seen this slower behavior even manually. Was there some update to getty or systemd that could have caused a slow down here?
[8 Apr 2018 08:10:09] <okurz> tacit: So actually I would consider a product bug but try to remove our needles nevertheless. We can remove the needle but it might break some tests.

Reproducible

Fails since (at least) Build 20180407-3 (current job)

Expected result

  • AC: There should be no needle with tag "text-login" anymore but only tty$nr-specific ones as we did already for SLE to prevent premature matching

Tasks

  1. Remove all needles with tag "text-login"
  2. Fix the tests that use that needle tag.

Suggestions

  • Find current uses of the generic needles, e.g. "text-login-20160812" using https://openqa.opensuse.org/admin/needles
  • Make sure there exist valid tty$nr-selected needles for all cases where text-login is matched on
  • Delete the too generic needles
  • Crosscheck carefully that all tests on o3 do not fail because of now deleted needles where there is no tty$nr-replacement.

Further details

Always latest result in this scenario: latest

See #33097 for the work that has been done for the corresponding SLE needles.


Related issues

Related to openQA Tests - action #32926: [sle][functional][y][hyperv][medium] avoid typing usernam... Resolved 08/03/2018 22/05/2018
Related to openQA Tests - action #33019: [opensuse][functional][sporadic][u][medium] Frequent fail... Resolved 09/03/2018 22/05/2018
Related to openQA Tests - action #36126: [functional][u] post_fail_hook matches on "text_login_roo... Resolved 14/05/2018
Related to openQA Tests - action #57281: [sle][Migration][SLE15SP2] test fails in orphaned_package... Resolved 24/09/2019
Blocks openQA Tests - action #34339: [sle][functional][medium][u] test fails in consoletest_s... Blocked 05/04/2018
Blocked by openQA Tests - action #32746: [sle][tools][remote-backends][hard] Incomplete job becaus... Resolved 05/03/2018

History

#1 Updated by okurz almost 2 years ago

  • Description updated (diff)

#2 Updated by okurz almost 2 years ago

  • Due date changed from 22/05/2018 to 08/05/2018
  • Priority changed from Normal to High

https://openqa.opensuse.org/tests/653247#step/firefox_audio/6 shows what I think is the same problem, in the post_fail_hook the needle text-login-20170427 is matching but we are not yet on the right tty, expecting tty5, seeing tty4. This prevents a proper analysis if the firefox_audio module itself is at fault or the system is bogged down due to something else.

#3 Updated by okurz almost 2 years ago

  • Blocks action #33019: [opensuse][functional][sporadic][u][medium] Frequent fails in firefox_audio, firefox not started added

#4 Updated by mgriessmeier almost 2 years ago

  • Subject changed from [functional][opensuse][u]too early matching in too generic needle text-login-20160812 to [functional][opensuse][u][medium] too early matching in too generic needle text-login-20160812
  • Status changed from New to Workable

#5 Updated by cwh almost 2 years ago

  • Difficulty set to medium

#6 Updated by SLindoMansilla almost 2 years ago

  • Status changed from Workable to In Progress
  • Assignee set to SLindoMansilla

#8 Updated by SLindoMansilla almost 2 years ago

Scenarios that would fail without a needle "text-login".

Found on https://openqa.suse.de/admin/needles (search: text-login)

OSD
- virt-pvusb-sles12sp2-fv-on-developing-xen@64bit-ipmi
- gi-guest_sles12sp2-on-host-developing-kvm@64bit-ipmi
- gi-guest_developing-on-host_sles12sp1-kvm@64bit-ipmi
- prj2_host_upgrade_sles12sp3_to_developing_xen@64bit-ipmi

Found on https://openqa.opensuse.org/admin/needles (search: text-login)

O3
- zdup-13.1-gnome@64bit setup_zdup/3 Needle created, but need to track (tty-6-selected-20180425) (verified)
- Same job, but in other module consoletest_setup/33 (verified)
- textmode@64bit Needle created, but need to track (tty4-selected-20180425) (verified)

#9 Updated by SLindoMansilla almost 2 years ago

Adapt code to not use generic needle text-login: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4942

#10 Updated by okurz almost 2 years ago

  • Related to action #34339: [sle][functional][medium][u] test fails in consoletest_setup - generic text login matching too early, causing mistyping -- was " login prompt not ready to type - missing keys while typing login" added

#11 Updated by SLindoMansilla almost 2 years ago

  • Related to action #32926: [sle][functional][y][hyperv][medium] avoid typing username before switched tty (was: test fails in yast2_i - (mising needles?, rather too low timeout for hyperv) for Installation Report succesful) added

#12 Updated by mgriessmeier almost 2 years ago

  • Due date changed from 08/05/2018 to 22/05/2018

#13 Updated by okurz almost 2 years ago

Let's work on #32926 first, then continue here.

#14 Updated by SLindoMansilla almost 2 years ago

  • Status changed from In Progress to Workable

Waiting for https://progress.opensuse.org/issues/32926

and

Problems with IPMI machines:

  • openqaw4-sp.qa.suse.de
  • sonic-sp.qa.suse.de

szarate and nsinger are investigating the issue.

#15 Updated by mgriessmeier almost 2 years ago

  • Due date changed from 22/05/2018 to 05/06/2018

dependency resolved, this one is workable now

#16 Updated by mgriessmeier almost 2 years ago

  • Blocks deleted (action #33019: [opensuse][functional][sporadic][u][medium] Frequent fails in firefox_audio, firefox not started)

#17 Updated by mgriessmeier almost 2 years ago

  • Related to action #33019: [opensuse][functional][sporadic][u][medium] Frequent fails in firefox_audio, firefox not started added

#18 Updated by jorauch almost 2 years ago

  • Related to deleted (action #34339: [sle][functional][medium][u] test fails in consoletest_setup - generic text login matching too early, causing mistyping -- was " login prompt not ready to type - missing keys while typing login")

#19 Updated by jorauch almost 2 years ago

  • Blocks action #34339: [sle][functional][medium][u] test fails in consoletest_setup - generic text login matching too early, causing mistyping -- was " login prompt not ready to type - missing keys while typing login" added

#20 Updated by mgriessmeier almost 2 years ago

  • Due date changed from 05/06/2018 to 19/06/2018
  • Assignee deleted (SLindoMansilla)
  • Target version changed from Milestone 16 to Milestone 17

#21 Updated by zluo almost 2 years ago

  • Assignee set to zluo

take over...

#22 Updated by zluo almost 2 years ago

  • Status changed from Workable to In Progress

check and replace assert_screen 'text_login' by wait_boot and select_console 'root-console':

sle-15-Installer-DVD-x86_64-Build665.2-external_iso@64bit
copy_iso_to_external_drive
post_run_hook contains assert_screen 'text_login' which needs to be changed

http://e13.suse.de/tests/2731#step/copy_iso_to_external_drive/29
looks good.

#24 Updated by zluo almost 2 years ago

PR for tests/virt_autotest/reboot_and_wait_up.pm:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5208

#25 Updated by zluo almost 2 years ago

#26 Updated by zluo almost 2 years ago

checked tests/console/sle15_workarounds.pm as well, keep text-login because workaround is still needed.

#27 Updated by zluo almost 2 years ago

lib/susedistribution.pm: assert_screen(["text-logged-in-$user", "text-login"], 60); no need to change, just keep it

#28 Updated by zluo almost 2 years ago

lib/susedistribution.pm: assert_screen "text-login", 10;

is used in the live-cd accessing the "install-shell". That is used for example here:

https://openqa.opensuse.org/tests/688993#step/logs_from_installation_system/1

We keep if (LIVECD) part as well.

#29 Updated by zluo almost 2 years ago

lib/susedistribution.pm: # Accept 'text-login' by default

remove the old comment.

#31 Updated by okurz almost 2 years ago

zluo wrote:

checked tests/console/sle15_workarounds.pm as well, keep text-login because workaround is still needed.

zluo wrote:

lib/susedistribution.pm: assert_screen(["text-logged-in-$user", "text-login"], 60); no need to change, just keep it

Please adress my comment in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5206#issuecomment-396524370

#32 Updated by zluo almost 2 years ago

tried to change tests/virt_autotest/reboot_and_wait_up.pm: assert_screen("text-login", 600); by using:

$self->wait_boot();
select_console 'root-console';

And change wait_boot function in lib/opensusebasetest.pm to avoid BACKEND impi run into GRUB:

elsif ((!(check_var('VIRSH_VMM_FAMILY', 'xen') && check_var('VIRSH_VMM_TYPE', 'linux') && check_var('BACKEND', 'svirt'))) || !(check_var('BACKEND', 'ipmi')))

This doesn't help:

http://e13.suse.de/tests/2749#step/reboot_and_wait_up_normal/6

#33 Updated by zluo almost 2 years ago

at moment the ipmi worker is not stable and it skips many times my test runs :(

http://e13.suse.de/tests/2833

#34 Updated by okurz almost 2 years ago

  • Target version changed from Milestone 17 to Milestone 17

#35 Updated by zluo almost 2 years ago

for ipmi use root-ssh instead of root-console and use PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5236 for another try:

http://e13.suse.de/tests/2840

#36 Updated by zluo almost 2 years ago

http://e13.suse.de/tests/2842

it seems to have same issue for:

at /usr/lib/os-autoinst/backend/baseclass.pm line 80.
backend::baseclass::die_handler('The console isn\'t responding correctly. Maybe half-open sock...') called at /usr/lib/os-autoinst/backend/baseclass.pm line 241
eval {...} called at /usr/lib/os-autoinst/backend/baseclass.pm line 156
backend::baseclass::run_capture_loop('backend::ipmi=HASH(0x699cc40)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 129
backend::baseclass::run('backend::ipmi=HASH(0x699cc40)', 5, 8) called at /usr/lib/os-autoinst/backend/driver.pm line 85
backend::driver::start('backend::driver=HASH(0x6493890)') called at /usr/lib/os-autoinst/backend/driver.pm line 48
backend::driver::new('backend::driver', 'ipmi') called at /usr/bin/isotovideo line 236
main::init_backend() called at /usr/bin/isotovideo line 305

#37 Updated by zluo almost 2 years ago

the issue reported in (https://progress.opensuse.org/issues/32746) needs to revolved at first if we want to make changes in reboot_and_wait_up.pm

#38 Updated by mgriessmeier almost 2 years ago

  • Due date changed from 19/06/2018 to 03/07/2018
  • Status changed from In Progress to Blocked

so it's basically blocked by https://progress.opensuse.org/issues/32746

#39 Updated by mgriessmeier almost 2 years ago

  • Blocked by action #32746: [sle][tools][remote-backends][hard] Incomplete job because console isn't responding correctly. Half-open socket on IPMI added

#40 Updated by zluo almost 2 years ago

  • % Done changed from 0 to 80

I think we are fine with the status of this ticket for now. Only reboot_and_wait_up.pm cannot be handled as we wish. So this can be worked on later when #32746 got resolved.

#41 Updated by mgriessmeier over 1 year ago

  • Due date changed from 03/07/2018 to 17/07/2018

#42 Updated by mgriessmeier over 1 year ago

  • Due date changed from 17/07/2018 to 31/07/2018
  • Status changed from Blocked to Feedback

#43 Updated by okurz over 1 year ago

  • Due date deleted (31/07/2018)
  • Status changed from Feedback to Workable
  • Assignee deleted (zluo)
  • Target version changed from Milestone 17 to Milestone 21+

ok, so we have covered a lot already. As we understood in SR zluo is not currently working on this at the moment. We should revisit the situation in some months again.

#44 Updated by okurz over 1 year ago

  • Priority changed from High to Normal

still valid but not "High" prio.

#45 Updated by okurz about 1 year ago

  • Target version changed from Milestone 21+ to future

#46 Updated by okurz about 1 year ago

I found that there is "linux-login" as well as "text-login". I assume both are equivalent and should be merged.

$ git grep '\(text\|linux\)-login\>'
lib/migration.pm:    if (!check_screen('linux-login', 200)) {
lib/opensusebasetest.pm:        push @tags, 'linux-login'                     if get_var('KEEP_GRUB_TIMEOUT');    # Also wait for linux-login if grub timeout was not disabled
lib/opensusebasetest.pm:        elsif (!match_has_tag("grub2") and !match_has_tag('linux-login')) {
lib/opensusebasetest.pm:            my $failneedle = get_var('KEEP_GRUB_TIMEOUT') ? 'linux-login' : 'grub2';
lib/opensusebasetest.pm:        my $textmode_needles = [qw(linux-login emergency-shell emergency-mode)];
lib/susedistribution.pm:            assert_screen "text-login", 10;
lib/susedistribution.pm:        assert_screen(["text-logged-in-$user", "text-login"], 60);
tests/boot/boot_from_pxe.pm:        assert_screen 'linux-login', 100;
tests/console/copy_iso_to_external_drive.pm:    assert_screen "text-login";
tests/console/sle15_workarounds.pm:        assert_screen(["tty2-selected", 'text-login', 'text-logged-in-root', 'generic-desktop']);
tests/installation/boot_into_snapshot.pm:    assert_screen 'linux-login', 200;
tests/installation/first_boot.pm:        assert_screen('linux-login', $boot_timeout) unless check_var('ARCH', 's390x');
tests/installation/grub_test.pm:    # If grub timeout was not disabled, we wait for linux-login instead
tests/installation/grub_test.pm:    my $tag = get_var('KEEP_GRUB_TIMEOUT') ? 'linux-login' : 'grub2';
tests/installation/reboot_eject_cd.pm:    assert_screen('text-login');
tests/jeos/firstrun.pm:    assert_screen_with_soft_timeout('linux-login', timeout => 1000, soft_timeout => 300, bugref => 'bsc#1077007');
tests/locale/keymap_or_locale.pm:        assert_screen([qw(linux-login cleared-console)]);
tests/locale/keymap_or_locale.pm:        assert_screen([qw(linux-login cleared-console)]);
tests/qa_automation/kernel_kexec.pm:    assert_screen('linux-login', 300);
tests/qam-minimal/install_update.pm:            assert_screen 'linux-login';
tests/sles4sap/netweaver_ascs_install.pm:              text-login linux-login started-x-displaymanager-info)
tests/virt_autotest/login_console.pm:    send_key_until_needlematch(['linux-login', 'virttest-displaymanager'], 'ret', $timeout / 5, 5);
tests/virt_autotest/reboot_and_wait_up.pm:        assert_screen("text-login", 600);
tests/virtualization/prepare.pm:        assert_screen 'linux-login', 600;
tests/virtualization/xen/patch_and_reboot.pm:    send_key_until_needlematch('text-login', 'ret', 360, 5);

So let's take a look for places where we have "text-login" and decide which are ok to keep or not

$ git grep '\<text-login\>'
lib/susedistribution.pm:            assert_screen "text-login", 10;

lib/susedistribution.pm:        assert_screen(["text-logged-in-$user", "text-login"], 60);
tests/console/copy_iso_to_external_drive.pm:    assert_screen "text-login";
tests/console/sle15_workarounds.pm:        assert_screen(["tty2-selected", 'text-login', 'text-logged-in-root', 'generic-desktop']);
tests/installation/reboot_eject_cd.pm:    assert_screen('text-login');
tests/sles4sap/netweaver_ascs_install.pm:              text-login linux-login started-x-displaymanager-info)
tests/virt_autotest/reboot_and_wait_up.pm:        assert_screen("text-login", 600);
  • 1. [ ] lib/susedistribution.pm: assert_screen "text-login", 10;

is used in the live-cd accessing the "install-shell". That is used for example here:
https://openqa.opensuse.org/tests/688993#step/logs_from_installation_system/1

Could you please try to just remove that if (LIVECD) part?

  • 2. [x] lib/susedistribution.pm: # Accept 'text-login' by default

should just be removed -> done with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5219

  • 3. [ ] lib/susedistribution.pm: assert_screen(["text-logged-in-$user", "text-login"], 60);

is within $type eq 'ssh' and I do not know if we still need that. Please take a look in used needles in scenarios using ssh and check where this part of the code is covered.

  • 4. [ ] tests/console/sle15_workarounds.pm: assert_screen(["tty2-selected", 'text-login', 'text-logged-in-root', 'generic-desktop']);

Please discuss with test module maintainer @rwx788 how to proceed with this test module. The bug for which this code is used as workaround is still valid and open. One solution I could think if is renaming the test module. Maybe we do not need the text-login needle here but can just work with the tty2-selected needle although explicitly mentioning the tty number is not so good. Please look into extracting the current console number from the root console object.

  • 5. [ ] tests/installation/reboot_eject_cd.pm: assert_screen('text-login');

is used for example in https://openqa.opensuse.org/tests/677992#step/reboot_eject_cd/1 . I guess you can simply replace text-login by tty3-selected because one line above there is a send_key 'ctrl-alt-f3'

  • 6. [ ] tests/sles4sap/netweaver_ascs_install.pm: text-login linux-login started-x-displaymanager-info)

should be replaced by wait_boot

  • 7. [ ] tests/virt_autotest/reboot_and_wait_up.pm: assert_screen("text-login", 600);

same, as already discussed yesterday, to be replaced by wait_boot

#47 Updated by zluo 8 months ago

  • Status changed from Workable to In Progress
  • Assignee set to zluo
  • Target version changed from future to Milestone 26

take over and checking current status.

#48 Updated by zluo 8 months ago

#49 Updated by zluo 8 months ago

  • Status changed from In Progress to Rejected

this is not an issue anymore:
http://f40.suse.de/tests/4869#next_previous

#50 Updated by okurz 8 months ago

  • Status changed from Rejected to In Progress

Why do you think this is not an issue anymore when we still haven't fulfilled the requirements we have set on ourselves? Please read the ticket description and comments again.

#51 Updated by zluo 8 months ago

  • Assignee deleted (zluo)

since the original issue doesn't happen anymore and this is not a priority at moment, I won't work on it.

#52 Updated by zluo 8 months ago

  • Status changed from In Progress to Workable

#53 Updated by mgriessmeier 7 months ago

  • Status changed from Workable to New
  • Priority changed from Normal to High
  • Target version changed from Milestone 26 to Milestone 27

discuss in grooming again ...

#54 Updated by SLindoMansilla 7 months ago

  • Description updated (diff)
  • Status changed from New to Workable
  • Priority changed from High to Normal

#55 Updated by SLindoMansilla 7 months ago

  • Estimated time set to 42.00

#56 Updated by mgriessmeier 7 months ago

  • Target version changed from Milestone 27 to Milestone 28

#57 Updated by okurz 6 months ago

  • Related to action #36126: [functional][u] post_fail_hook matches on "text_login_root" before actual tty switch and therefore never logs in added

#58 Updated by zluo 4 months ago

  • Status changed from Workable to In Progress
  • Assignee set to zluo

take over and check if this is a similar issue because I'm working on https://progress.opensuse.org/issues/36126

#59 Updated by zluo 4 months ago

found still some text-login:
text-login-20130711.json: "text-login"
text-login-20160812.json: "text-login"
text-login-20170427.json: "text-login"
text-login-20170504.json: "text-login"

We still have following test modules:

lib/sles4sap.pm: text-login linux-login started-x-displaymanager-info)
lib/susedistribution.pm: assert_screen "text-login", 10;
lib/susedistribution.pm: assert_screen(["text-logged-in-$user", "text-login"], 60);
tests/console/copy_iso_to_external_drive.pm: assert_screen "text-login";
tests/console/sle15_workarounds.pm: assert_screen(["tty2-selected", 'text-login', 'text-logged-in-root', 'generic-desktop']);
tests/installation/reboot_eject_cd.pm: assert_screen('text-login');
tests/systemd_testsuite/test_09_issue_2691.pm: send_key_until_needlematch('text-login', 'ret', 360, 5);
tests/virt_autotest/reboot_and_wait_up.pm: assert_screen("text-login", 600);

above test modules are tested for sle but not for opensuse.

To compare with #46 and consider to how to process with this ticket again.

#60 Updated by zluo 4 months ago

found:

lib/caasp.pm: assert_screen 'linux-login-casp', 150;
lib/caasp.pm: # Don't match linux-login-casp twice
lib/caasp.pm: assert_screen [qw(grub2 linux-login-casp)], 150;
lib/locale.pm: assert_screen([qw(linux-login cleared-console)]);
lib/locale.pm: assert_screen(has_ttys() ? 'linux-login' : 'cleared_console');
lib/migration.pm:# Assert screen 'linux-login' with 200s
lib/migration.pm: if (!check_screen('linux-login', 200)) {
lib/opensusebasetest.pm: my $textmode_needles = [qw(linux-login emergency-shell emergency-mode)];
lib/opensusebasetest.pm: # 2nd stage of autoyast can be considered as linux-login
lib/sles4sap.pm: text-login linux-login started-x-displaymanager-info)
lib/systemd_testsuite_test.pm: assert_screen('linux-login', 300);
lib/transactional.pm: assert_screen 'linux-login', 200;
tests/autoyast/installation.pm: push @needles, 'linux-login-casp' if is_caasp;
tests/autoyast/installation.pm: || match_has_tag('linux-login-casp')
tests/boot/boot_from_pxe.pm: assert_screen 'linux-login', 100;
tests/boot/boot_linuxrc.pm: assert_screen([qw(linux-login displaymanager generic-desktop)], 180);
tests/boot/grub2_test.pm: assert_screen 'linux-login', 200;
tests/boot/grub2_test.pm: assert_screen 'linux-login', 300;
tests/boot/grub2_test.pm: assert_screen 'linux-login', 200;
tests/boot/grub2_test.pm: assert_screen 'linux-login', 100;
tests/installation/boot_into_snapshot.pm: assert_screen 'linux-login', 200;
tests/installation/first_boot.pm: assert_screen('linux-login', $boot_timeout) unless check_var('ARCH', 's390x');
tests/installation/ipxe_install.pm: assert_screen('linux-login', 1800);
tests/jeos/firstrun.pm: assert_screen [qw(linux-login reached-power-off)], 1000;
tests/jeos/prepare_firstboot.pm: assert_screen('linux-login', 300);
tests/kernel/boot_ltp.pm: assert_screen('linux-login', 1800);
tests/kernel/build_git_kernel.pm: assert_screen('linux-login', 1800);
tests/kernel/kernel_kexec.pm: # clear console to prevent linux-login to match before reboot
tests/kernel/kernel_kexec.pm: assert_screen('linux-login', 300);
tests/kernel/mellanox_config.pm: assert_screen('linux-login', 1800);
tests/kernel_performance/run_perf_case.pm: assert_screen('linux-login', $timeout);
tests/kiwi_images_test/kiwi_boot.pm: assert_screen('linux-login', 1200);
tests/kiwi_images_test/kiwi_boot.pm: assert_screen('linux-login', 1000);
tests/kiwi_images_test/login_reboot.pm: assert_screen('linux-login', 120);
tests/kiwi_images_test/validate_build.pm: assert_screen('linux-login', 120);
tests/locale/keymap_or_locale.pm: assert_screen([qw(linux-login cleared-console)]);
tests/qam-minimal/install_update.pm: assert_screen 'linux-login', 90;
tests/virt_autotest/login_console.pm: send_key_until_needlematch(['linux-login', 'virttest-displaymanager'], 'ret', $counter, $timeout);
tests/virt_autotest/login_console.pm: assert_screen('will-linux-login', $timeout);
tests/virt_autotest/login_console.pm: last if (check_screen([qw(linux-login virttest-displaymanager)], 60));
tests/virtualization/prepare.pm: assert_screen 'linux-login', 600;

tag 'linux-login' found from needles:

linux-login-20140916.json: "linux-login"
linux-login-autoyast-20150724.json: "linux-login"
linux-login-autoyast-20160822.json: "linux-login"
linux-login-autoyast-20191126.json: "linux-login",
linux-login-kubic-20180201.json: "linux-login-casp"
linux-login-kubic-20190408.json: "linux-login-casp"
linux-login-kubic-20190415.json: "linux-login-casp"
linux-login-kubic-20191031.json: "linux-login-casp"

#61 Updated by zluo 4 months ago

http://f40.suse.de/tests/5546#step/zypper_up/1 (JeOS-for-kvm-and-xen-x86_64-Build17.7-jeos@64bit_virtio-2G)
http://f40.suse.de/tests/5547#step/zypper_up/1 (Tumbleweed-NET-x86_64-Build20191126-textmode@64bit)
http://f40.suse.de/tests/5548#step/zypper_up/1 (opensuse-15.1-DVD-Updates-x86_64-Build20191127-2-textmode@64bit)
http://f40.suse.de/tests/5549#step/zypper_up/1 (opensuse-15.0-DVD-Updates-x86_64-Build20191127-2-textmode@64bit)

show that root-console-20170321 is used. We don't have problem at moment for opensuse. And we can delete text-login from opensuse needle repository.

#63 Updated by zluo 4 months ago

http://f40.suse.de/tests/5551#step/zypper_up/41 shows network issue and triggers later post_fail_hook: http://f40.suse.de/tests/5551#step/zypper_up/41

tty5-selected is then correctly use now. So we are safe to remove text-login.

#66 Updated by zluo 4 months ago

Now checking linux-login tag which is used by opensuse:

lib/caasp.pm: assert_screen 'linux-login-casp', 150;
lib/caasp.pm: # Don't match linux-login-casp twice
lib/caasp.pm: assert_screen [qw(grub2 linux-login-casp)], 150;
lib/locale.pm: assert_screen([qw(linux-login cleared-console)]);
lib/locale.pm: assert_screen(has_ttys() ? 'linux-login' : 'cleared_console');
lib/migration.pm:# Assert screen 'linux-login' with 200s
lib/migration.pm: if (!check_screen('linux-login', 200)) {
lib/opensusebasetest.pm: my $textmode_needles = [qw(linux-login emergency-shell emergency-mode)];
lib/opensusebasetest.pm: # 2nd stage of autoyast can be considered as linux-login
lib/sles4sap.pm: text-login linux-login started-x-displaymanager-info)
lib/systemd_testsuite_test.pm: assert_screen('linux-login', 300);
lib/transactional.pm: assert_screen 'linux-login', 200;
tests/autoyast/installation.pm: push @needles, 'linux-login-casp' if is_caasp;
tests/autoyast/installation.pm: || match_has_tag('linux-login-casp')
tests/boot/boot_from_pxe.pm: assert_screen 'linux-login', 100;
tests/boot/boot_linuxrc.pm: assert_screen([qw(linux-login displaymanager generic-desktop)], 180);
tests/boot/grub2_test.pm: assert_screen 'linux-login', 200;
tests/boot/grub2_test.pm: assert_screen 'linux-login', 300;
tests/boot/grub2_test.pm: assert_screen 'linux-login', 200;
tests/boot/grub2_test.pm: assert_screen 'linux-login', 100;
tests/installation/boot_into_snapshot.pm: assert_screen 'linux-login', 200;
tests/installation/first_boot.pm: assert_screen('linux-login', $boot_timeout) unless check_var('ARCH', 's390x');
tests/installation/ipxe_install.pm: assert_screen('linux-login', 1800);
tests/jeos/firstrun.pm: assert_screen [qw(linux-login reached-power-off)], 1000;
tests/jeos/prepare_firstboot.pm: assert_screen('linux-login', 300);
tests/kernel/boot_ltp.pm: assert_screen('linux-login', 1800);
tests/kernel/build_git_kernel.pm: assert_screen('linux-login', 1800);
tests/kernel/kernel_kexec.pm: # clear console to prevent linux-login to match before reboot
tests/kernel/kernel_kexec.pm: assert_screen('linux-login', 300);
tests/kernel/mellanox_config.pm: assert_screen('linux-login', 1800);
tests/kernel_performance/run_perf_case.pm: assert_screen('linux-login', $timeout);
tests/kiwi_images_test/kiwi_boot.pm: assert_screen('linux-login', 1200);
tests/kiwi_images_test/kiwi_boot.pm: assert_screen('linux-login', 1000);
tests/kiwi_images_test/login_reboot.pm: assert_screen('linux-login', 120);
tests/kiwi_images_test/validate_build.pm: assert_screen('linux-login', 120);
tests/locale/keymap_or_locale.pm: assert_screen([qw(linux-login cleared-console)]);
tests/qam-minimal/install_update.pm: assert_screen 'linux-login', 90;
tests/virt_autotest/login_console.pm: send_key_until_needlematch(['linux-login', 'virttest-displaymanager'], 'ret', $counter, $timeout);
tests/virt_autotest/login_console.pm: assert_screen('will-linux-login', $timeout);
tests/virt_autotest/login_console.pm: last if (check_screen([qw(linux-login virttest-displaymanager)], 60));
tests/virtualization/prepare.pm: assert_screen 'linux-login', 600;

found: first_boot.pm, firstrun.pm, keymap_or_locale.pm are valid for opensuse tw.
Need to check these test modules for sle and create required needle 'tty$Nr-selected' as well!

#67 Updated by zluo 4 months ago

http://f40.suse.de/tests/5567 done for first_boot and keymap_or_locale.

#68 Updated by zluo 4 months ago

http://f40.suse.de/tests/5574#step/firstrun/7 shows successful test run with a new needle 'tty6-selected'.

#70 Updated by zluo 4 months ago

for sle-15-SP2-Online-x86_64-Build93.1-gnome@64bit:
http://f40.suse.de/tests/5576
first_boot and keymap_or_locale are not effected by code changes:

elsif ((get_var('DESKTOP', '') =~ /textmode|serverro/) || get_var('BOOT_TO_SNAPSHOT')) {
my $tty = get_root_console_tty;
assert_screen("tty$tty-selected", $boot_timeout) unless check_var('ARCH', 's390x');
return;
}
and 
assert_screen([qw("tty$tty-selected" cleared-console)]);

#71 Updated by dimstar 4 months ago

  • Priority changed from Normal to Urgent

zluo wrote:

needle PR: https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/619

This has been merged and the live installer in Tumbleweed are now broken

https://openqa.opensuse.org/tests/1099159#next_previous

#73 Updated by zluo 4 months ago

okay, will check this then.

#75 Updated by zluo 4 months ago

test_09_issue_2691:
http://f40.suse.de/tests/5585#step/test_09_issue_2691/1 shows that needle match and it has tags: text-login and tty6-selected

#76 Updated by zluo 4 months ago

installation_overview.pm still uses tag 'text-login', need to check where this is coming from.
it comes from install-shell - > text-login. Change it to tty2-selected:

http://f40.suse.de/tests/5589#step/installation_overview/2

#77 Updated by zluo 4 months ago

clone https://openqa.suse.de/tests/3641764 for change 'tty1-selected' in tests/virt_autotest/reboot_and_wait_up.pm

#80 Updated by zluo 4 months ago

  • Status changed from In Progress to Feedback
  • Priority changed from Urgent to High

#81 Updated by mgriessmeier 3 months ago

  • Target version changed from Milestone 28 to Milestone 30

needs to be discussed offline

#82 Updated by okurz 3 months ago

  • Related to action #57281: [sle][Migration][SLE15SP2] test fails in orphaned_packages_check - switch to tty failed added

#83 Updated by zluo about 1 month ago

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9048 merged now.

checking now old PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9034

first_boot.pm got changed and it uses now function wait_boot_past_bootloader and verify this change by test runs:

diff --git a/lib/opensusebasetest.pm b/lib/opensusebasetest.pm
index 08648c884..6b5ee125b 100644
--- a/lib/opensusebasetest.pm
+++ b/lib/opensusebasetest.pm
@@ -806,7 +806,7 @@ sub wait_boot_textmode {
return if check_var('ARCH', 's390x');
my $ready_time       = $args{ready_time};
-    my $textmode_needles = [qw(linux-login emergency-shell emergency-mode)];
+    my $textmode_needles = [qw(tty1-selected emergency-shell emergency-mode)];
# 2nd stage of autoyast can be considered as linux-login
push @{$textmode_needles}, 'autoyast-init-second-stage' if get_var('AUTOYAST');
# Soft-fail for user_defined_snapshot in extra_tests_on_gnome and extra_tests_on_gnome_on_ppc

#85 Updated by xlai about 1 month ago

zluo wrote:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9034 updated now.

@zluo @okurz Would really appreciate if we can be involved in solution discussion and PR review earlier if next time such big influence changes are made. Our tests in latest build 134.1 fail a lot at matching tty1-selected in https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=134.1&groupid=263.

#86 Updated by okurz about 1 month ago

xlai wrote:

zluo wrote:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9034 updated now.


@zluo @okurz Would really appreciate if we can be involved in solution discussion and PR review earlier if next time such big influence changes are made.

I didn't do any changes within the last year. I only participated in review and merged a revert so far. The above mentioned PR was already reverted in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9473 so maybe a retrigger should work. In general you can participate in the solution discussion since two years (!) but I did not see any comment by you in this ticket. Also the PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9034 was merged after being open for more than 2 months (!) so you could have looked into the PR for a very long time already. Expecting anyone would decide when to personally ping you or not on indvidual pull requests is an approach which can be used in exceptions but it isn't going to scale well. IMHO the pull request being open for two months is more than enough. However I grant you that the PR has a lacking description and was ill-prepared and is missing a lot of verification runs. Early review can all help with this.

EDIT: sorry, meant to cross-check mainly IPMI, not (only) ssh

#87 Updated by xlai about 1 month ago

okurz wrote:

xlai wrote:

zluo wrote:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9034 updated now.


@zluo @okurz Would really appreciate if we can be involved in solution discussion and PR review earlier if next time such big influence changes are made.


I didn't do any changes within the last year. I only participated in review and merged a revert so far. The above mentioned PR was already reverted in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9473 so maybe a retrigger should work.

I at you because you are the creator of this ticket and I thought you created the "expected results" and "tasks" , which showed the solution. Hope you do not mind :).

In general you can participate in the solution discussion since two years (!) but I did not see any comment by you in this ticket. Also the PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9034 was merged after being open for more than 2 months (!) so you could have looked into the PR for a very long time already. Expecting anyone would decide when to personally ping you or not on indvidual pull requests is an approach which can be used in exceptions but it isn't going to scale well. IMHO the pull request being open for two months is more than enough. However I grant you that the PR has a lacking description and was ill-prepared and is missing a lot of verification runs. Early review can all help with this.

Yes, you are right. But looking into all open PRs and reviewing them will take a lot of time. It is quite expensive to be aware of the big changes from this PR review way. It is a common challenge for most openqa test code maintainers.

@zluo @slindomansilla Introducing a check for "tty1" is IMHO a pretty bad idea and far from the original idea of this ticket. Our tests should stay generic and keep systems in mind that do not have tty's, e.g. remote systems where we login over ssh.

I am afraid so. There should be a better way to solve the original ticket. Now virtualization tests are blocked a lot. And the influence of replacing such general needle "linux-login" should be much bigger.

#88 Updated by okurz about 1 month ago

I at you because you are the creator of this ticket and I thought you created the "expected results" and "tasks" , which showed the solution. Hope you do not mind :).

I think you did good. Just wanted to put in perspective what my current impact is :)

Yes, you are right. But looking into all open PRs and reviewing them will take a lot of time. It is quite expensive to be aware of the big changes from this PR review way. It is a common challenge for most openqa test code maintainers.

I understand that. Basically I see three different choices as of now:

  1. Review PRs, at best before merge
  2. Investigate which test changes can explain a test fail. This is also now provided in the "Investigation" tab of each test

#89 Updated by SLindoMansilla about 1 month ago

Suggestion:

  • create a new needle covering the ttyS1 terminal with tag 'ttyS1-selected'.
  • Or revert this specific change and leave text-login as general needle.

#90 Updated by zluo about 1 month ago

since PR for linux-login got reverted, and I created now a new PR to fix issue:

https://openqa.nue.suse.com/tests/3853054

This should be safe from now with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9487 (still need verification run).

#91 Updated by zluo about 1 month ago

https://openqa.suse.de/tests/3858631#step/reboot_and_wait_up_normal/4 looks good. The new needle created contains 2 tags: ttys1-selected and tty1-selected

#93 Updated by zluo about 1 month ago

since my PR got reverted again, we can do this now:

Or revert this specific change and leave text-login as general needle

However with a new needle it fixed the issue, but actually this not easy to cover all cases which text-login still used widely for other tests/team, maybe this is better, don't to try this for now till we have a better idea.

See example:
https://149.44.176.58/tests/3869862#step/reboot_and_wait_up_normal/4

#94 Updated by zluo about 1 month ago

  • Status changed from Feedback to Workable
  • Assignee deleted (zluo)

since my PR got reverted again, we can do this now:

Or revert this specific change and leave text-login as general needle

However with a new needle it fixed the issue, but actually this not easy to cover all cases where text-login is still used widely. Maybe this is better, don't to try this for now till we have a "final" solution.

#95 Updated by xlai about 1 month ago

This change still fails 3 tests in virtualization-acceptance job group, https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=136.2&groupid=263. It also fails several tests in virtualization-milestone group, although some are still scheduled, https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=136.2&groupid=264.

So due to this change, we can not deliver the public beta candidate acceptance test report in time.

If a full support can not be done, I strongly suggest a full revert of all PRs, rather than partially revert and partially keep. @mgressimier @zluo, how do you think?

#96 Updated by SLindoMansilla about 1 month ago

  • Status changed from Workable to New

To be discussed again

#97 Updated by SLindoMansilla about 1 month ago

  • Priority changed from High to Normal

#98 Updated by okurz 24 days ago

  • Category changed from Bugs in existing tests to Enhancement to existing tests

Actually the original issue is fixed since long but the enhancement as mentioned in the subject is still valid.

Also available in: Atom PDF