action #56045

[functional][u][sporadic] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm

Added by JRivrain 6 months ago. Updated 7 days ago.

Status:NewStart date:28/08/2019
Priority:HighDue date:
Assignee:-% Done:

0%

Category:Bugs in existing testsEstimated time:42.00 hours
Target version:SUSE QA tests - Milestone 30
Difficulty:
Duration:

Description

Observation

Test times out at dasd_configure. This happens not every time but quite often, since at least build 294.

openQA test in scenario sle-12-SP5-Server-DVD-s390x-btrfs_libstorage@s390x-zVM-hsi-l2 fails in
bootloader_start

Acceptance Criteria

AC1: remove workaround and set device down again after bug is fixed

Reproducible

Fails since (at least) Build 0296

Expected result

Last good: 0296 (or more recent)

Further details

Always latest result in this scenario: latest


Related issues

Blocks openQA Tests - action #48434: [functional][u] test Tumbleweed s390x again Blocked 26/02/2019

History

#1 Updated by mgriessmeier 6 months ago

  • Subject changed from [functional] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm to [functional][u] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm
  • Status changed from New to Workable
  • Target version set to Milestone 27

#2 Updated by SLindoMansilla 6 months ago

  • Estimated time set to 42.00

It looks like a problem with the serial device.
I was able to reproduce it on my local openQA setup at home 4 of 4 times.

#3 Updated by SLindoMansilla 6 months ago

Sometimes it works: https://openqa.suse.de/tests/3321190#step/bootloader_s390/36
Sometimes it fails later: https://openqa.suse.de/tests/3321149#step/bootloader_s390/46
Does someone know how to debug what's happening on the serial device?
As this kind of failure always became the 'hated by all', we should document this for future references.

#4 Updated by SLindoMansilla 6 months ago

  • Blocks action #48434: [functional][u] test Tumbleweed s390x again added

#5 Updated by SLindoMansilla 6 months ago

  • Status changed from Workable to In Progress
  • Assignee set to SLindoMansilla

#8 Updated by mgriessmeier 5 months ago

  • Status changed from Rejected to In Progress
  • Target version changed from Milestone 27 to Milestone 28

#9 Updated by mgriessmeier 5 months ago

  • Assignee changed from SLindoMansilla to mgriessmeier

#11 Updated by mgriessmeier 5 months ago

32 passed, 18 failed -> 36% failure rate in https://openqa.suse.de/tests/overview?version=12-SP5&build=mgriessmeier_poo56045&distri=sle

I will try to improve it - or better find the root cause and fix it

#12 Updated by SLindoMansilla 5 months ago

Thanks! I actually have no idea how to fix it.

#13 Updated by mgriessmeier 5 months ago

filed product bug https://bugzilla.suse.com/show_bug.cgi?id=1151394 against 12SP5, also happens on 15SP2 (will clarify if I should clone it)
will implement softfail workaround tomorrow

#14 Updated by okurz 5 months ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: btrfs_libstorage@s390x-zVM-hsi-l2
https://openqa.suse.de/tests/3418584

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

#16 Updated by mgriessmeier 4 months ago

  • Description updated (diff)
  • Status changed from In Progress to Blocked

added AC to remove workaround again

#17 Updated by mgriessmeier about 1 month ago

  • Target version changed from Milestone 28 to Milestone 30

Workaround is still triggered sometimes
revisit next week

#18 Updated by mgriessmeier about 1 month ago

  • Assignee changed from mgriessmeier to SLindoMansilla

#19 Updated by SLindoMansilla about 1 month ago

Trying Berthold's suggestion: https://bugzilla.opensuse.org/show_bug.cgi?id=1156047#c5

hardened_usercopy=off

#20 Updated by SLindoMansilla about 1 month ago

  • Status changed from Blocked to In Progress

#22 Updated by SLindoMansilla about 1 month ago

Verified on OSD: https://openqa.suse.de/tests/3789298

Added WORKAROUND_BUGS=bsc1156047 to machine settings in OSD

  • s390x-zVM-ctc
  • s390x-zVM-hsi-l2
  • s390x-zVM-hsi-l3
  • s390x-zVM-vswitch-l2

#23 Updated by SLindoMansilla about 1 month ago

Verified on O3: https://openqa.opensuse.org/tests/1144907#step/bootloader_s390/45

Added WORKAROUND_BUGS=bsc1156047 to machine settings in OSD
- s390x

(we should change the machine name to match their OSD equivalent)

#25 Updated by mgriessmeier about 1 month ago

hmm guess the setting was added to the machine after the job ran

#26 Updated by SLindoMansilla about 1 month ago

JRivrain wrote:

Seems to still happen :( https://openqa.suse.de/tests/3803881#step/bootloader_start/48

Sorry, my mistake, I didn't notice that there was a second page on the machines view with the machine this job uses.
Setting added.

#28 Updated by SLindoMansilla 29 days ago

mgriessmeier wrote:

still happens :(
https://openqa.suse.de/tests/3822768#step/bootloader_s390/47

I realized that it is sporadic.

#29 Updated by okurz 15 days ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: textmode@s390x-zVM-vswitch-l3
https://openqa.suse.de/tests/3869904

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

#30 Updated by openqa_review 15 days ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: textmode@s390x-zVM-vswitch-l3
https://openqa.suse.de/tests/3869904

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

#31 Updated by SLindoMansilla 7 days ago

  • Subject changed from [functional][u] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm to [functional][u][sporadic] command 'dasd_configure 0.0.0150 0' timed out at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/installation/bootloader_s390.pm
  • Description updated (diff)
  • Status changed from In Progress to New
  • Assignee deleted (SLindoMansilla)

Also available in: Atom PDF