Project

General

Profile

Actions

action #13914

open

[qe-core][functional][ipmi] wait_serial does not get expected output because ipmi console connection is closed

Added by xlai over 7 years ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
2016-09-27
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Test failed due to wait_serial does not get output. From serial0.txt, the ipmi session was already closed due to "excess errors received"

Failure step:
https://openqa.suse.de/tests/587781#step/install_package/4

Build link:
https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2141&groupid=46

Serial output link:
https://openqa.suse.de/tests/587781/file/serial0.txt

Key serial output errors:

[�[0;32m OK �[0m] Started Serial Getty on ttyS1.
[�[0;32m OK �[0m] Started Serial Getty on hvc0.
     Starting X Display Manager...
[�[0;32m OK �[0m] Started Getty on tty1.
[�[0;32m OK �[0m] Reached target Login Prompts.
[�[0;32m OK �[0m] Started /etc/init.d/after.local Compatibility.
[�[0;32m OK �[0m] Started Load dom0 backend drivers.
     Starting The Xen xenstore...
[SOL established]
[error received]: excess errors received
[closing the connection]

Related issues 4 (2 open2 closed)

Related to openQA Project - action #18144: [tools] restart ipmi management controller before every ipmi jobResolvedRBrownSUSE2017-03-29

Actions
Related to openQA Project - action #23404: Serial output gets lostResolved2017-08-16

Actions
Related to openQA Tests - coordination #44843: [qe-core][functional][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module)New2018-12-13

Actions
Blocked by openQA Tests - action #34699: [qe-core][functional][ipmi] access to serial log during installationNew2018-04-10

Actions
Actions #1

Updated by okurz over 7 years ago

hm, what do you think could be the potential error sources? Is it the IPMI itself or network connection or related to test or backend changes or maybe the product itself?

Actions #2

Updated by xlai over 7 years ago

I do not have any proof yet. But I strongly suspect the ipmi performance under high test pressure.

Actions #3

Updated by okurz over 7 years ago

  • Category set to Bugs in existing tests
Actions #4

Updated by okurz about 7 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gi-guest_sles11sp4-on-host_sles12sp3-xen
http://openqa.suse.de/tests/781640

Actions #5

Updated by okurz about 7 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gi-guest_sles11sp4-on-host_sles12sp3-xen
http://openqa.suse.de/tests/781640

Actions #6

Updated by okurz almost 7 years ago

  • Related to action #18144: [tools] restart ipmi management controller before every ipmi job added
Actions #7

Updated by okurz almost 7 years ago

https://openqa.suse.de/tests/964225#step/install_and_reboot/10 looks related, serial0.txt says

IKG0B-162-
DIE Can't close(GLOB(0x610ab00)) filehandle: 'No child processes' at /usr/lib/os-autoinst/backend/baseclass.pm line 267

 at /usr/lib/os-autoinst/backend/baseclass.pm line 73.
    backend::baseclass::die_handler('autodie::exception=HASH(0x6200770)') called at (eval 1258) line 75

so same?

https://openqa.suse.de/tests/966840#step/zypper_lr/2 looks similar or related. serial0.txt shows the output but wait_serial does not show the expected output. As it looks like in the same job wait_serial failed in before in an intermediate step, e.g. https://openqa.suse.de/tests/966840#step/logpackages/4

@coolo is this related to your latest work on ipmi+ssh?

Actions #8

Updated by okurz almost 7 years ago

Retriggered job https://openqa.suse.de/tests/967718 shows the same symptoms, no serial output recorded in the wait_serial calls

Actions #9

Updated by okurz almost 7 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gnome@64bit-ipmi
https://openqa.suse.de/tests/1002264

Actions #10

Updated by okurz over 6 years ago

Actions #12

Updated by xlai over 6 years ago

This issue is absolutely ipmi itself unstability issue, has nothing to do with test itself. If relevant, then we can only say that high work pressure on ipmi may lead to this much more.

Actions #13

Updated by okurz over 6 years ago

  • Description updated (diff)
  • Category changed from Bugs in existing tests to Infrastructure
  • Priority changed from Normal to High

updated description format. IMHO it's an "infrastructure" issue. I don't see anything done wrong in the test code itself. Of course, if we can not find a more stable solution in any IMPI worker that means the ticket is about workarounds in the test code or the backend itself.

Actions #14

Updated by okurz almost 6 years ago

  • Related to action #34699: [qe-core][functional][ipmi] access to serial log during installation added
Actions #15

Updated by okurz over 5 years ago

  • Related to coordination #44843: [qe-core][functional][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module) added
Actions #16

Updated by okurz about 5 years ago

  • Subject changed from [ipmi] wait_serial does not get expected output because ipmi console connection is closed to [functional][u][ipmi] wait_serial does not get expected output because ipmi console connection is closed
  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version set to future

waiting for #34699 first

Actions #17

Updated by okurz almost 5 years ago

  • Assignee changed from okurz to mgriessmeier

Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.

Actions #18

Updated by SLindoMansilla almost 4 years ago

  • Related to deleted (action #34699: [qe-core][functional][ipmi] access to serial log during installation)
Actions #19

Updated by SLindoMansilla almost 4 years ago

  • Blocked by action #34699: [qe-core][functional][ipmi] access to serial log during installation added
Actions #20

Updated by SLindoMansilla almost 4 years ago

  • Assignee changed from mgriessmeier to SLindoMansilla
Actions #21

Updated by tjyrinki_suse over 3 years ago

  • Subject changed from [functional][u][ipmi] wait_serial does not get expected output because ipmi console connection is closed to [qe-core][functional][ipmi] wait_serial does not get expected output because ipmi console connection is closed
Actions #22

Updated by SLindoMansilla about 3 years ago

  • Status changed from Blocked to New
  • Assignee deleted (SLindoMansilla)

No time to work on this :(

Actions #23

Updated by okurz over 2 years ago

  • Priority changed from High to Normal

This ticket was set to "High" priority but was not updated within 120 days which is 4 times the period of the SLO for "High" tickets (30 days) as described on https://progress.opensuse.org/projects/openqatests/wiki/Wiki#SLOs-service-level-objectives . The ticket will be set to the next lower priority of "Normal".

Actions #24

Updated by slo-gin over 1 year ago

This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.

Actions

Also available in: Atom PDF