Project

General

Profile

action #56045

[qe-core][functional][sporadic] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm

Added by JRivrain over 2 years ago. Updated 10 months ago.

Status:
Rejected
Priority:
High
Category:
Bugs in existing tests
Target version:
SUSE QA - Milestone 30
Start date:
2019-08-28
Due date:
% Done:

0%

Estimated time:
42.00 h
Difficulty:

Description

Observation

Test times out at dasd_configure. This happens not every time but quite often, since at least build 294.

openQA test in scenario sle-12-SP5-Server-DVD-s390x-btrfs_libstorage@s390x-zVM-hsi-l2 fails in
bootloader_start

Acceptance Criteria

AC1: remove workaround and set device down again after bug is fixed

Reproducible

Fails since (at least) Build 0296

Expected result

Last good: 0296 (or more recent)

Further details

Always latest result in this scenario: latest

History

#1 Updated by mgriessmeier over 2 years ago

  • Subject changed from [functional] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm to [functional][u] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm
  • Status changed from New to Workable
  • Target version set to Milestone 27

#2 Updated by SLindoMansilla over 2 years ago

  • Estimated time set to 42.00 h

It looks like a problem with the serial device.
I was able to reproduce it on my local openQA setup at home 4 of 4 times.

#3 Updated by SLindoMansilla over 2 years ago

Sometimes it works: https://openqa.suse.de/tests/3321190#step/bootloader_s390/36
Sometimes it fails later: https://openqa.suse.de/tests/3321149#step/bootloader_s390/46
Does someone know how to debug what's happening on the serial device?
As this kind of failure always became the 'hated by all', we should document this for future references.

#4 Updated by SLindoMansilla over 2 years ago

  • Blocks action #48434: [functional][u] test Tumbleweed s390x again added

#5 Updated by SLindoMansilla over 2 years ago

  • Status changed from Workable to In Progress
  • Assignee set to SLindoMansilla

#8 Updated by mgriessmeier over 2 years ago

  • Status changed from Rejected to In Progress
  • Target version changed from Milestone 27 to Milestone 28

#9 Updated by mgriessmeier over 2 years ago

  • Assignee changed from SLindoMansilla to mgriessmeier

#11 Updated by mgriessmeier over 2 years ago

32 passed, 18 failed -> 36% failure rate in https://openqa.suse.de/tests/overview?version=12-SP5&build=mgriessmeier_poo56045&distri=sle

I will try to improve it - or better find the root cause and fix it

#12 Updated by SLindoMansilla over 2 years ago

Thanks! I actually have no idea how to fix it.

#13 Updated by mgriessmeier over 2 years ago

filed product bug https://bugzilla.suse.com/show_bug.cgi?id=1151394 against 12SP5, also happens on 15SP2 (will clarify if I should clone it)
will implement softfail workaround tomorrow

#14 Updated by okurz over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: btrfs_libstorage@s390x-zVM-hsi-l2
https://openqa.suse.de/tests/3418584

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#16 Updated by mgriessmeier about 2 years ago

  • Description updated (diff)
  • Status changed from In Progress to Blocked

added AC to remove workaround again

#17 Updated by mgriessmeier about 2 years ago

  • Target version changed from Milestone 28 to Milestone 30

Workaround is still triggered sometimes
revisit next week

#18 Updated by mgriessmeier about 2 years ago

  • Assignee changed from mgriessmeier to SLindoMansilla

#19 Updated by SLindoMansilla about 2 years ago

Trying Berthold's suggestion: https://bugzilla.opensuse.org/show_bug.cgi?id=1156047#c5

hardened_usercopy=off

#20 Updated by SLindoMansilla about 2 years ago

  • Status changed from Blocked to In Progress

#22 Updated by SLindoMansilla about 2 years ago

Verified on OSD: https://openqa.suse.de/tests/3789298

Added WORKAROUND_BUGS=bsc1156047 to machine settings in OSD

  • s390x-zVM-ctc
  • s390x-zVM-hsi-l2
  • s390x-zVM-hsi-l3
  • s390x-zVM-vswitch-l2

#23 Updated by SLindoMansilla about 2 years ago

Verified on O3: https://openqa.opensuse.org/tests/1144907#step/bootloader_s390/45

Added WORKAROUND_BUGS=bsc1156047 to machine settings in OSD

  • s390x

(we should change the machine name to match their OSD equivalent)

#25 Updated by mgriessmeier almost 2 years ago

hmm guess the setting was added to the machine after the job ran

#26 Updated by SLindoMansilla almost 2 years ago

JRivrain wrote:

Seems to still happen :( https://openqa.suse.de/tests/3803881#step/bootloader_start/48

Sorry, my mistake, I didn't notice that there was a second page on the machines view with the machine this job uses.
Setting added.

#28 Updated by SLindoMansilla almost 2 years ago

mgriessmeier wrote:

still happens :(
https://openqa.suse.de/tests/3822768#step/bootloader_s390/47

I realized that it is sporadic.

#29 Updated by okurz almost 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: textmode@s390x-zVM-vswitch-l3
https://openqa.suse.de/tests/3869904

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#30 Updated by openqa_review almost 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: textmode@s390x-zVM-vswitch-l3
https://openqa.suse.de/tests/3869904

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#31 Updated by SLindoMansilla almost 2 years ago

  • Subject changed from [functional][u] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm to [functional][u][sporadic] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm
  • Description updated (diff)
  • Status changed from In Progress to New
  • Assignee deleted (SLindoMansilla)

#32 Updated by SLindoMansilla almost 2 years ago

  • Assignee set to mgriessmeier

#33 Updated by mgriessmeier almost 2 years ago

  • Status changed from New to Blocked

there was improved debug output PR by Jozef: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9664

still considering this as blocked by bsc#1156047: https://bugzilla.opensuse.org/show_bug.cgi?id=1156047

#34 Updated by SLindoMansilla almost 2 years ago

  • Blocks deleted (action #48434: [functional][u] test Tumbleweed s390x again)

#36 Updated by okurz almost 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@s390x-zVM-vswitch-l3
https://openqa.suse.de/tests/4028628

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#37 Updated by okurz almost 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@s390x-zVM-vswitch-l3
https://openqa.suse.de/tests/4028628

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#38 Updated by openqa_review almost 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@s390x-zVM-vswitch-l3
https://openqa.suse.de/tests/4093600

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#39 Updated by okurz over 1 year ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: allpatterns@s390x-zVM-vswitch-l3
https://openqa.suse.de/tests/4170908

To prevent further reminder comments one of the following options should be followed:

  1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
  2. The openQA job group is moved to "Released"
  3. The label in the openQA scenario is removed

#40 Updated by SLindoMansilla over 1 year ago

  • Status changed from Blocked to Workable
  • Assignee changed from mgriessmeier to SLindoMansilla

change the command from assert_script_run to script_run and if return code > 0 => record_soft_failure

#41 Updated by tjyrinki_suse about 1 year ago

  • Subject changed from [functional][u][sporadic] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm to [qe-core][functional][sporadic] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm

#42 Updated by SLindoMansilla 10 months ago

  • Status changed from Workable to Rejected

Bug fixed for SLE15-SP3 installer.

For release products:

Also available in: Atom PDF