Project

General

Profile

Actions

action #16488

closed

[sles][functional][tools][s390x] zVM: test fails in first_boot to reconnect to s390x host

Added by okurz about 7 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Start date:
2017-02-06
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-12-SP3-Server-DVD-s390x-migration_offline_sle12sp2_zVM@s390x-zVM-Upgrade fails to reconnect in https://openqa.suse.de/tests/758636

logfile:

[…]
          'Welcome to SUSE Linux Enterprise Server 12 SP3 Alpha2 (s390x) - Kernel 4.4.44-1-'
        ];

01:40:31.9723 Debug: /var/lib/openqa/share/tests/sle/tests/installation/reconnect_s390.pm:40 called testapi::select_console
01:40:31.9725 22793 <<< testapi::select_console(testapi_console='iucvconn')
01:40:32.0300 22794 Could not connect to root@s390vsl152.suse.de, Retry
01:40:42.0717 22794 Could not connect to root@s390vsl152.suse.de, Retry
01:40:52.0730 22794 Could not connect to root@s390vsl152.suse.de, Retry
01:41:02.0743 22794 Could not connect to root@s390vsl152.suse.de, Retry
01:41:12.0755 22794 Could not connect to root@s390vsl152.suse.de, Retry
DIE Failed to login to root@s390vsl152.suse.de at /usr/lib/os-autoinst/backend/baseclass.pm line 995.

 at /usr/lib/os-autoinst/backend/baseclass.pm line 73.
    backend::baseclass::die_handler('Failed to login to root@s390vsl152.suse.de at /usr/lib/os-aut...') called at /usr/lib/os-autoinst/backend/baseclass.pm line 995
    backend::baseclass::new_ssh_connection('backend::s390x=HASH(0x62b2398)', 'hostname', 's390vsl152.suse.de', 'password', 'nots3cr3t', 'username', 'root') called at /usr/lib/os-autoinst/consoles/sshIucvconn.pm line 40
    consoles::sshIucvconn::activate('consoles::sshIucvconn=HASH(0x53d7268)') called at /usr/lib/os-autoinst/consoles/console.pm line 74
[…]

Reproducible

Fails since (at least) Build 0234 (current job)

Expected result

Last good: 0229 (or more recent)

Further details

Always latest result in this scenario: latest


Related issues 4 (0 open4 closed)

Related to openQA Tests - action #18890: [s390x][zVM]test fails to reconnect after installation -> incompleteResolvedokurz2017-05-01

Actions
Related to openQA Tests - action #25414: [sle][functional][s390x][zvm] improve debugging for zvm only shows black screen in reconnect_s390 and error is hidden in "expect_3270" outputResolvedokurz

Actions
Related to openQA Tests - action #25584: [sle][functional][s390x][sporadic][medium] backend connection lost during "partitioning_warnings", vnc stalled, etc -> should turn into failResolvedmgriessmeier2017-09-272018-03-27

Actions
Blocked by openQA Tests - action #19262: [functional][s390] openQA fails to reconnect to SUT after rebootResolvedokurz2017-05-19

Actions
Actions #1

Updated by mgriessmeier about 7 years ago

  • Priority changed from Normal to Low

lowering priority - didn't see this for a while now

Actions #2

Updated by mgriessmeier about 7 years ago

  • Subject changed from s390x zVM: test fails in first_boot to reconnect so s390x host to [sles][functional][tools][s390x] zVM: test fails in first_boot to reconnect so s390x host
Actions #3

Updated by mgriessmeier about 7 years ago

  • Status changed from New to Rejected

reopen if you see this again

Actions #4

Updated by mgriessmeier almost 7 years ago

  • Status changed from Rejected to In Progress
Actions #5

Updated by okurz almost 7 years ago

  • Priority changed from Low to High

Is https://openqa.suse.de/tests/956659 the same? https://openqa.suse.de/tests/956659/file/autoinst-log.txt shows

$VAR1 = bless( {
                 'queue' => [
                              'Storage cleared - system reset.                                                 ',
                              'Booting default (grub2)                                                         '
                            ]
               }, 'Thread::Queue' );

11:00:04.1358 24981 considering VNC stalled, no update for 4.29 seconds
DIE socket does not exist. Probably your backend instance could not start or died. at /usr/lib/os-autoinst/consoles/VNC.pm line 881.

 at /usr/lib/os-autoinst/backend/baseclass.pm line 73.
    backend::baseclass::die_handler('socket does not exist. Probably your backend instance could n...') called at /usr/lib/os-autoinst/consoles/VNC.pm line 801

so it tries to boot grub2 but then VNC stalled and what does the DIE mean and where does it come frome? Again IMHO a case of not really helpful output.

Also https://openqa.suse.de/tests/956599 seems to fail for the same or similar reason. There is no mention of "grub2" in this case. Do we need to bump a timeout or something here?

Actions #6

Updated by okurz almost 7 years ago

  • Related to action #18890: [s390x][zVM]test fails to reconnect after installation -> incomplete added
Actions #7

Updated by mgriessmeier almost 7 years ago

  • Assignee deleted (mgriessmeier)

Happens way too sporadic for me to spend time with investigating this deeper - I blame temporary network issues, feel free to take over
(without checking in detail how often it fails, so you could still proof me wrong with some statistic)

Workaround: restart

Actions #8

Updated by okurz almost 7 years ago

  • Blocked by action #19262: [functional][s390] openQA fails to reconnect to SUT after reboot added
Actions #9

Updated by okurz almost 7 years ago

  • Status changed from In Progress to Feedback
  • Assignee set to okurz

blocked by #19262

Actions #10

Updated by okurz almost 7 years ago

  • Status changed from Feedback to Resolved

Related tickets closed, code change. Maybe done. Reopen otherwise.

Actions #11

Updated by okurz almost 7 years ago

  • Status changed from Resolved to In Progress
  • Priority changed from High to Normal

Ok so we are back to the original issue: http://opeth.suse.de/tests/5368/file/autoinst-log.txt

Actions #12

Updated by okurz over 6 years ago

  • Assignee deleted (okurz)
Actions #13

Updated by okurz over 6 years ago

  • Subject changed from [sles][functional][tools][s390x] zVM: test fails in first_boot to reconnect so s390x host to [sles][functional][tools][s390x] zVM: test fails in first_boot to reconnect to s390x host
Actions #14

Updated by okurz over 6 years ago

  • Related to action #25414: [sle][functional][s390x][zvm] improve debugging for zvm only shows black screen in reconnect_s390 and error is hidden in "expect_3270" output added
Actions #15

Updated by okurz over 6 years ago

  • Related to action #25584: [sle][functional][s390x][sporadic][medium] backend connection lost during "partitioning_warnings", vnc stalled, etc -> should turn into fail added
Actions #16

Updated by okurz over 6 years ago

  • Target version set to Milestone 14
Actions #17

Updated by riafarov about 6 years ago

  • Status changed from In Progress to Workable
Actions #18

Updated by okurz about 6 years ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz

I think we have this covered by improving the automatic failure investigation on first boot for s390x.

Actions

Also available in: Atom PDF