Project

General

Profile

Actions

action #25376

closed

coordination #28504: [sle][functional][epic] crypt lvm for SLE 12-SP3, SLE 12-SP4 and SLE 15

[sle][functional][ppc64le][hard] test fails in grub_test - stuck in grub screen, expected encrypted password prompt (was: boots completely)

Added by jorauch over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Urgent
Category:
Bugs in existing tests
Start date:
2017-09-18
Due date:
2017-12-06
% Done:

0%

Estimated time:
Difficulty:

Description

Acceptance criteria

AC1: Product bugs related to cryptlvm are properly filed for SLE 15 on aarch64, ppc64 and x86_64

Tasks

  1. Determine and file product bugs related to cryptlvm for SLE 15 on aarch64, ppc64 and x86_64

Observation

Example for ppc: https://openqa.suse.de/tests/1173256#step/grub_test/1

Looks like the test boots straight to the login screen but expects to see Grub.
Maybe some needles are missing?

SLE 12-SP3 aarch

SLE 12-SP3 ppc

SLE 12-SP3 x86

SLE 15 349.1 aarch

SLE 15 349.1 ppc

SLE 15 349.1 x86

openQA test in scenario sle-15-Leanos-DVD-aarch64-yast_no_self_update@aarch64 fails in
grub_test

Reproducible

Fails since (at least) Build 260.4 (current job)

Acceptance criteria

  • AC1: Tests are green on all architectures again
  • AC2: Changes do not break tests for older versions (SLE12)

Tasks

  • Adapt test flow to architecture (aarch64 1x password, ppc64le 2x password)
  • Verify for all architectures

Expected result

Last good: 257.3 (or more recent)

All tests are either working or breaking in a product bug

Further details

Always latest result in this scenario: latest


Related issues 5 (0 open5 closed)

Related to openQA Tests - action #25388: [sle][functional][sle15][aarch64]test fails in grub_testResolvedJERiveraMoya2017-09-182017-11-08

Actions
Related to openQA Tests - action #25472: [sles][functional][aarch64][sporadic] test fails in grub_test: openQA detects stall (cannot find grub2 needle) but system boots completely as seen in serial0.txtRejected2017-09-21

Actions
Related to openQA Tests - action #27397: [sle][functional][openqareview] Verification that AArch64 is able to boot from encrypted LVM - Password prompt appears on serial console instead of plymouth screenResolvedJERiveraMoya2017-11-082018-01-30

Actions
Blocked by openQA Tests - action #25658: [sle][functional][migration][opensuse][virtualization]Increase/disable timeout of initial grub menue to ensure tests do not miss itResolvedJERiveraMoya2017-09-292017-10-25

Actions
Blocked by openQA Tests - action #27082: [sle][functional] test fails in scc_registration - Since the beta version, modules are hidden by defaultResolvedSLindoMansilla2017-10-27

Actions
Actions #1

Updated by okurz over 6 years ago

  • Priority changed from Normal to High

the aarch64 test seems to have missed the grub menue completely, looking at the video. The ppc64le test is tricky because no videos are recorded on ppc64le by default. I recommend to manually retrigger one with NOVIDEO=0 and monitor that one.

Actions #2

Updated by okurz over 6 years ago

  • Subject changed from [sle][functional][aarch64][ppc64le] test fails in grub_test - boots completely to [sle][functional][ppc64le] test fails in grub_test - boots completely
Actions #3

Updated by okurz over 6 years ago

  • Related to action #25388: [sle][functional][sle15][aarch64]test fails in grub_test added
Actions #4

Updated by okurz over 6 years ago

  • Target version set to Milestone 11
Actions #5

Updated by mgriessmeier over 6 years ago

  • Related to action #25472: [sles][functional][aarch64][sporadic] test fails in grub_test: openQA detects stall (cannot find grub2 needle) but system boots completely as seen in serial0.txt added
Actions #6

Updated by okurz over 6 years ago

  • Blocked by action #25658: [sle][functional][migration][opensuse][virtualization]Increase/disable timeout of initial grub menue to ensure tests do not miss it added
Actions #7

Updated by jorauch over 6 years ago

Also happens in boot_encrypt
https://openqa.suse.de/tests/1224378#comments

Actions #8

Updated by okurz over 6 years ago

hm, ok. so that's interesting because we now have "disable_grub_timeout" so we should not miss the grub screen. So this is a different issue then.

Actions #9

Updated by okurz over 6 years ago

  • Due date set to 2017-11-08
Actions #10

Updated by okurz over 6 years ago

  • Subject changed from [sle][functional][ppc64le] test fails in grub_test - boots completely to [sle][functional][ppc64le] test fails in grub_test - stuck in grub screen, expected encrypted password prompt (was: boots completely)
Actions #11

Updated by JERiveraMoya over 6 years ago

  • Status changed from New to In Progress
  • Assignee set to JERiveraMoya
Actions #13

Updated by SLindoMansilla over 6 years ago

  • Blocked by action #27082: [sle][functional] test fails in scc_registration - Since the beta version, modules are hidden by default added
Actions #14

Updated by okurz over 6 years ago

PR merged, triggered one job manually:

openqa_clone_job_osd 1235824 _GROUP=0 TEST=okurz_crosscheck_cryptlvm_minimal_x

Created job #1239194: sle-15-Installer-DVD-ppc64le-Build321.5-cryptlvm_minimal_x@ppc64le -> https://openqa.suse.de/t1239194

Actions #15

Updated by JERiveraMoya over 6 years ago

Test moved forward, but lines https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/installation/grub_test.pm#L76-L79 make it stuck again, checking for the grub screen again. Should be removed now with grub timeout disabled?

Actions #16

Updated by JERiveraMoya over 6 years ago

After discussion with the team about the use of this test and also with the developer of the offending lines it seems reasonable to exclude them instead of delete them.
PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3837

Actions #18

Updated by JERiveraMoya over 6 years ago

I got to a solution that is suppose to unblock staging and also fix the ppc64. PR modified (not closed, sorry for the previous confusion). It was successful with 64bit. Asked mgriessmeier for verification run in ppc. The problem that the grub appears before the passphrase need to be further discussed with developers I think.

Actions #19

Updated by JERiveraMoya over 6 years ago

It still failing for ppc64: https://openqa.suse.de/tests/1242988# but at least unblock failures for 64bit: https://openqa.suse.de/tests/1242989#

Actions #20

Updated by JERiveraMoya over 6 years ago

We can see failures in grub_test and boot_crypt tests. Sometimes the first does actions that the second should do due to the following differences in behavior for each architectures:

cryptlvm tests for SLE15:

Comparing with SLE-12-SP3:

Not sure what is the right approach to continue, meaning the correct workflow for each architecture. Set ticket on feedback.

Actions #21

Updated by JERiveraMoya over 6 years ago

  • Status changed from In Progress to Feedback
Actions #22

Updated by JERiveraMoya over 6 years ago

  • Status changed from Feedback to In Progress
Actions #24

Updated by JERiveraMoya over 6 years ago

Verification failed because grub_test is typing passphrase for ppc. Excluding this part only for ppc: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3872
New verification seems successful: https://openqa.suse.de/tests/1244836#step/boot_encrypt/3

Actions #25

Updated by SLindoMansilla over 6 years ago

  • Related to action #27397: [sle][functional][openqareview] Verification that AArch64 is able to boot from encrypted LVM - Password prompt appears on serial console instead of plymouth screen added
Actions #26

Updated by JERiveraMoya over 6 years ago

Current status of this ticket:
ppc64le:

This task need to be picked up by someone else due to is required local verification for ppc to consolidate logic in tests grub_test and boot_encrypt.
That could be handled in a different ticket due to the main fix of this ticket is done, that is to pass the grub screen (due to disabled grub timeout) or to be continued in this one (asking Oliver about it).

Last re-triggered of the build: https://progress.opensuse.org/issues/25854 asked for password once, so both jobs are successful.

Actions #27

Updated by okurz over 6 years ago

  • Due date changed from 2017-11-08 to 2017-11-22
  • Priority changed from High to Urgent
  • Target version changed from Milestone 11 to Milestone 12

We agree that this became a bigger issue now and we should work on this together. Scheduling for next sprint to re-evaluate.
We should understand the expected product behaviour, split the tasks, review open bugs, open fate issues, etc.

Actions #28

Updated by okurz over 6 years ago

  • Due date changed from 2017-11-22 to 2017-12-06
  • Assignee deleted (JERiveraMoya)

really need to take a look in more detail and not wait for a single person to get lost here :)

Actions #29

Updated by JERiveraMoya over 6 years ago

Regarding comments in related bug https://bugzilla.suse.com/show_bug.cgi?id=1062806#c25 seems to be reasonable that the first passphrase does not appear in ppc64le due to a dedicated /boot partition is present in this architecture outside the encrypted volume. In aarch64 and x86_64 there is no dedicated /boot partition, therefore grub prompts passphrase as it wants to load kernel and initrd from the LVM encrypted partition.
Current jobs related with encryption are successful so it might not be needed further actions from our side on ppc in this ticket:
https://openqa.suse.de/tests/1268728
https://openqa.suse.de/tests/1268722

Actions #30

Updated by jorauch over 6 years ago

  • Subject changed from [sle][functional][ppc64le] test fails in grub_test - stuck in grub screen, expected encrypted password prompt (was: boots completely) to [sle][functional][ppc64le][hard] test fails in grub_test - stuck in grub screen, expected encrypted password prompt (was: boots completely)
  • Description updated (diff)

What is the current state of this ticket, I see that it works for ppc, but ARM is still broken

Actions #31

Updated by jorauch over 6 years ago

  • Description updated (diff)
Actions #32

Updated by SLindoMansilla over 6 years ago

  • Assignee set to SLindoMansilla

Further investigation is needed here. Collecting important information here.

SLE 12-SP3 aarch
https://openqa.suse.de/tests/1057976#step/partitioning_lvm/4
https://openqa.suse.de/tests/1057976#step/grub_test/1 no password asked
https://openqa.suse.de/tests/1057976#step/boot_encrypt/1 password asked

SLE 12-SP3 ppc
https://openqa.suse.de/tests/1058033#step/partitioning_lvm/4
https://openqa.suse.de/tests/1058033#step/grub_test/1 password asked
https://openqa.suse.de/tests/1058033#step/boot_encrypt/1 password asked

SLE 12-SP3 x86
https://openqa.suse.de/tests/1058314#step/partitioning_lvm/4
https://openqa.suse.de/tests/1058314#step/grub_test/1 no password asked
https://openqa.suse.de/tests/1058314#step/boot_encrypt/1 password asked

SLE 15 349.1 aarch
https://openqa.suse.de/tests/1268571#step/partitioning_lvm/7
https://openqa.suse.de/tests/1268571#step/grub_test/1 password asked
N/A
• openQA test fails in boot_encrypt - AArch64 cannot boot from encrypted LVM - https://bugzilla.suse.com/show_bug.cgi?id=1062806
• passphrase for the password is asked in text, instead of gui on arm and x86_64 due to dropped plymouth - https://bugzilla.suse.com/show_bug.cgi?id=1067481

SLE 15 349.1 ppc
https://openqa.suse.de/tests/1268728#step/partitioning_lvm/7
https://openqa.suse.de/tests/1268728#step/grub_test/1 no password asked
https://openqa.suse.de/tests/1268728#step/boot_encrypt/1 password asked

SLE 15 349.1 x86
https://openqa.suse.de/tests/1269048#step/partitioning_lvm/7
https://openqa.suse.de/tests/1269048#step/grub_test/1 password asked
https://openqa.suse.de/tests/1269048#step/boot_encrypt/1 password asked
passphrase is asked twice when encrypt root partition on arm and x86_64 - https://bugzilla.suse.com/show_bug.cgi?id=1067479

Actions #33

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
  • Status changed from In Progress to Feedback

On aarch64 and x86_64 the behavior is expected because of the requirements:

But it is not wanted, so a bug is filed: https://bugzilla.suse.com/show_bug.cgi?id=1067479


On ppc64 the behavior is not wanted, because we want the boot partition to also be encrypted, so bug is filed: https://bugzilla.suse.com/show_bug.cgi?id=1070139


Then a new feature is expected for SLE 15 to only need to type the password once, based on: https://fate.suse.com/320901

Actions #34

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
Actions #35

Updated by SLindoMansilla over 6 years ago

  • Parent task set to #28504
Actions #36

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
  • Status changed from Feedback to Resolved

At the moment, all related product bugs are properly filed. There is no more work from QA to be done from this ticket.
Acceptance criteria for this planned ticket added and is already fulfilled.

Please, find next steps on parent task: https://progress.opensuse.org/issues/28504

Actions #37

Updated by SLindoMansilla over 6 years ago

  • Description updated (diff)
Actions

Also available in: Atom PDF