Project

General

Profile

action #13914

[qe-core][functional][ipmi] wait_serial does not get expected output because ipmi console connection is closed

Added by xlai about 5 years ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
2016-09-27
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Test failed due to wait_serial does not get output. From serial0.txt, the ipmi session was already closed due to "excess errors received"

Failure step:
https://openqa.suse.de/tests/587781#step/install_package/4

Build link:
https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2141&groupid=46

Serial output link:
https://openqa.suse.de/tests/587781/file/serial0.txt

Key serial output errors:

[�[0;32m OK �[0m] Started Serial Getty on ttyS1.
[�[0;32m OK �[0m] Started Serial Getty on hvc0.
     Starting X Display Manager...
[�[0;32m OK �[0m] Started Getty on tty1.
[�[0;32m OK �[0m] Reached target Login Prompts.
[�[0;32m OK �[0m] Started /etc/init.d/after.local Compatibility.
[�[0;32m OK �[0m] Started Load dom0 backend drivers.
     Starting The Xen xenstore...
[SOL established]
[error received]: excess errors received
[closing the connection]

Related issues

Related to openQA Project - action #18144: [tools] restart ipmi management controller before every ipmi jobResolved2017-03-29

Related to openQA Project - action #23404: Serial output gets lostResolved2017-08-16

Related to openQA Tests - coordination #44843: [qe-core][functional][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module)New2018-12-13

Blocked by openQA Tests - action #34699: [qe-core][functional][ipmi] access to serial log during installationNew2018-04-10

History

#1 Updated by okurz about 5 years ago

hm, what do you think could be the potential error sources? Is it the IPMI itself or network connection or related to test or backend changes or maybe the product itself?

#2 Updated by xlai about 5 years ago

I do not have any proof yet. But I strongly suspect the ipmi performance under high test pressure.

#3 Updated by okurz almost 5 years ago

  • Category set to Bugs in existing tests

#4 Updated by okurz over 4 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gi-guest_sles11sp4-on-host_sles12sp3-xen
http://openqa.suse.de/tests/781640

#5 Updated by okurz over 4 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gi-guest_sles11sp4-on-host_sles12sp3-xen
http://openqa.suse.de/tests/781640

#6 Updated by okurz over 4 years ago

  • Related to action #18144: [tools] restart ipmi management controller before every ipmi job added

#7 Updated by okurz over 4 years ago

https://openqa.suse.de/tests/964225#step/install_and_reboot/10 looks related, serial0.txt says

IKG0B-162-
DIE Can't close(GLOB(0x610ab00)) filehandle: 'No child processes' at /usr/lib/os-autoinst/backend/baseclass.pm line 267

 at /usr/lib/os-autoinst/backend/baseclass.pm line 73.
    backend::baseclass::die_handler('autodie::exception=HASH(0x6200770)') called at (eval 1258) line 75

so same?

https://openqa.suse.de/tests/966840#step/zypper_lr/2 looks similar or related. serial0.txt shows the output but wait_serial does not show the expected output. As it looks like in the same job wait_serial failed in before in an intermediate step, e.g. https://openqa.suse.de/tests/966840#step/logpackages/4

coolo is this related to your latest work on ipmi+ssh?

#8 Updated by okurz over 4 years ago

Retriggered job https://openqa.suse.de/tests/967718 shows the same symptoms, no serial output recorded in the wait_serial calls

#9 Updated by okurz over 4 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: gnome@64bit-ipmi
https://openqa.suse.de/tests/1002264

#10 Updated by okurz about 4 years ago

#12 Updated by xlai about 4 years ago

This issue is absolutely ipmi itself unstability issue, has nothing to do with test itself. If relevant, then we can only say that high work pressure on ipmi may lead to this much more.

#13 Updated by okurz about 4 years ago

  • Description updated (diff)
  • Category changed from Bugs in existing tests to Infrastructure
  • Priority changed from Normal to High

updated description format. IMHO it's an "infrastructure" issue. I don't see anything done wrong in the test code itself. Of course, if we can not find a more stable solution in any IMPI worker that means the ticket is about workarounds in the test code or the backend itself.

#14 Updated by okurz over 3 years ago

  • Related to action #34699: [qe-core][functional][ipmi] access to serial log during installation added

#15 Updated by okurz almost 3 years ago

  • Related to coordination #44843: [qe-core][functional][epic] Cleanup the use of serial-/virtio-/ssh-consoles in our tests (was: use $self->select_serial_terminal instead of checking IPMI in every module) added

#16 Updated by okurz almost 3 years ago

  • Subject changed from [ipmi] wait_serial does not get expected output because ipmi console connection is closed to [functional][u][ipmi] wait_serial does not get expected output because ipmi console connection is closed
  • Status changed from New to Blocked
  • Assignee set to okurz
  • Target version set to future

waiting for #34699 first

#17 Updated by okurz over 2 years ago

  • Assignee changed from okurz to mgriessmeier

Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.

#18 Updated by SLindoMansilla over 1 year ago

  • Related to deleted (action #34699: [qe-core][functional][ipmi] access to serial log during installation)

#19 Updated by SLindoMansilla over 1 year ago

  • Blocked by action #34699: [qe-core][functional][ipmi] access to serial log during installation added

#20 Updated by SLindoMansilla over 1 year ago

  • Assignee changed from mgriessmeier to SLindoMansilla

#21 Updated by tjyrinki_suse 12 months ago

  • Subject changed from [functional][u][ipmi] wait_serial does not get expected output because ipmi console connection is closed to [qe-core][functional][ipmi] wait_serial does not get expected output because ipmi console connection is closed

#22 Updated by SLindoMansilla 7 months ago

  • Status changed from Blocked to New
  • Assignee deleted (SLindoMansilla)

No time to work on this :(

#23 Updated by okurz about 2 months ago

  • Priority changed from High to Normal

This ticket was set to "High" priority but was not updated within 120 days which is 4 times the period of the SLO for "High" tickets (30 days) as described on https://progress.opensuse.org/projects/openqatests/wiki/Wiki#SLOs-service-level-objectives . The ticket will be set to the next lower priority of "Normal".

Also available in: Atom PDF