Project

General

Profile

action #13914

[functional][u][ipmi] wait_serial does not get expected output because ipmi console connection is closed

Added by xlai almost 4 years ago. Updated 3 months ago.

Status:
Blocked
Priority:
High
Category:
Infrastructure
Target version:
Start date:
2016-09-27
Due date:
% Done:

0%

Estimated time:
Difficulty:
Duration:

Description

Test failed due to wait_serial does not get output. From serial0.txt, the ipmi session was already closed due to "excess errors received"

Failure step:
https://openqa.suse.de/tests/587781#step/install_package/4

Build link:
https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2141&groupid=46

Serial output link:
https://openqa.suse.de/tests/587781/file/serial0.txt

Key serial output errors:

[�[0;32m OK �[0m] Started Serial Getty on ttyS1.
[�[0;32m OK �[0m] Started Serial Getty on hvc0.
     Starting X Display Manager...
[�[0;32m OK �[0m] Started Getty on tty1.
[�[0;32m OK �[0m] Reached target Login Prompts.
[�[0;32m OK �[0m] Started /etc/init.d/after.local Compatibility.
[�[0;32m OK �[0m] Started Load dom0 backend drivers.
     Starting The Xen xenstore...
[SOL established]
[error received]: excess errors received
[closing the connection]

Related issues

Related to openQA Project - action #18144: [tools] restart ipmi management controller before every ipmi jobResolved2017-03-29

Related to openQA Project - action #23404: Serial output gets lostResolved2017-08-16

Related to openQA Tests - action #44843: [functional][u][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module)Workable2018-12-13

Blocked by openQA Tests - action #34699: [functional][u][ipmi] access to serial log during installationBlocked2018-04-10

History

#1 Updated by okurz almost 4 years ago

hm, what do you think could be the potential error sources? Is it the IPMI itself or network connection or related to test or backend changes or maybe the product itself?

#2 Updated by xlai almost 4 years ago

I do not have any proof yet. But I strongly suspect the ipmi performance under high test pressure.

#3 Updated by okurz over 3 years ago

  • Category set to Bugs in existing tests

#4 Updated by okurz over 3 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gi-guest_sles11sp4-on-host_sles12sp3-xen
http://openqa.suse.de/tests/781640

#5 Updated by okurz over 3 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gi-guest_sles11sp4-on-host_sles12sp3-xen
http://openqa.suse.de/tests/781640

#6 Updated by okurz over 3 years ago

  • Related to action #18144: [tools] restart ipmi management controller before every ipmi job added

#7 Updated by okurz about 3 years ago

https://openqa.suse.de/tests/964225#step/install_and_reboot/10 looks related, serial0.txt says

IKG0B-162-
DIE Can't close(GLOB(0x610ab00)) filehandle: 'No child processes' at /usr/lib/os-autoinst/backend/baseclass.pm line 267

 at /usr/lib/os-autoinst/backend/baseclass.pm line 73.
    backend::baseclass::die_handler('autodie::exception=HASH(0x6200770)') called at (eval 1258) line 75

so same?

https://openqa.suse.de/tests/966840#step/zypper_lr/2 looks similar or related. serial0.txt shows the output but wait_serial does not show the expected output. As it looks like in the same job wait_serial failed in before in an intermediate step, e.g. https://openqa.suse.de/tests/966840#step/logpackages/4

coolo is this related to your latest work on ipmi+ssh?

#8 Updated by okurz about 3 years ago

Retriggered job https://openqa.suse.de/tests/967718 shows the same symptoms, no serial output recorded in the wait_serial calls

#9 Updated by okurz about 3 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gnome@64bit-ipmi
https://openqa.suse.de/tests/1002264

#10 Updated by okurz almost 3 years ago

#12 Updated by xlai almost 3 years ago

This issue is absolutely ipmi itself unstability issue, has nothing to do with test itself. If relevant, then we can only say that high work pressure on ipmi may lead to this much more.

#13 Updated by okurz almost 3 years ago

  • Description updated (diff)
  • Category changed from Bugs in existing tests to Infrastructure
  • Priority changed from Normal to High

updated description format. IMHO it's an "infrastructure" issue. I don't see anything done wrong in the test code itself. Of course, if we can not find a more stable solution in any IMPI worker that means the ticket is about workarounds in the test code or the backend itself.

#14 Updated by okurz over 2 years ago

  • Related to action #34699: [functional][u][ipmi] access to serial log during installation added

#15 Updated by okurz over 1 year ago

  • Related to action #44843: [functional][u][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module) added

#16 Updated by okurz over 1 year ago

  • Subject changed from [ipmi] wait_serial does not get expected output because ipmi console connection is closed to [functional][u][ipmi] wait_serial does not get expected output because ipmi console connection is closed
  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version set to future

waiting for #34699 first

#17 Updated by okurz about 1 year ago

  • Assignee changed from okurz to mgriessmeier

Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.

#18 Updated by SLindoMansilla 3 months ago

  • Related to deleted (action #34699: [functional][u][ipmi] access to serial log during installation)

#19 Updated by SLindoMansilla 3 months ago

  • Blocked by action #34699: [functional][u][ipmi] access to serial log during installation added

#20 Updated by SLindoMansilla 3 months ago

  • Assignee changed from mgriessmeier to SLindoMansilla

Also available in: Atom PDF