Project

General

Profile

action #30249

[sle][migration][sle15][ppc64le] test fails in install_and_reboot - failed by diskspace is exhausted

Added by JWSun over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Target version:
-
Start date:
2018-01-12
Due date:
% Done:

100%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-15-Installer-DVD-ppc64le-media_upgrade_sles12sp3+sdk_allpatterns@ppc64le fails in
install_and_reboot

Reproducible

Fails since (at least) Build 414.13

Expected result

Last good: (unknown) (or more recent)

Further details

Always latest result in this scenario: latest


Related issues

Is duplicate of openQA Infrastructure - action #30595: [ppc64le] Deploy bcache on tmpfs workersNew2018-01-22

History

#1 Updated by qmsu over 3 years ago

  • Subject changed from [sle][migration][sle15][ppc64le] test fails in install_and_reboot - failed by diskspace is exhausted to [sle][migration][sle15][ppc64le][medium] test fails in install_and_reboot - failed by diskspace is exhausted

#2 Updated by qmsu over 3 years ago

  • Subject changed from [sle][migration][sle15][ppc64le][medium] test fails in install_and_reboot - failed by diskspace is exhausted to [sle][migration][sle15][ppc64le] test fails in install_and_reboot - failed by diskspace is exhausted

#3 Updated by qmsu over 3 years ago

  • Priority changed from Normal to High

#4 Updated by JWSun over 3 years ago

More case failed by it
https://openqa.suse.de/tests/1399236

#5 Updated by qmsu over 3 years ago

  • Priority changed from High to Urgent

More tests are failed due to this issue.

#6 Updated by okurz over 3 years ago

  • Is duplicate of action #30595: [ppc64le] Deploy bcache on tmpfs workers added

#7 Updated by okurz over 3 years ago

  • Status changed from New to Rejected

see duplicatee #30595

#8 Updated by qmsu over 3 years ago

Likely another issue besides poo#30595, because test log showed 40G disk space of VM was exhausted

https://openqa.suse.de/tests/1380164/file/install_and_reboot-df.txt
/dev/vda3 40384512 39743552 6272 100% /mnt

I will check again with new test runs when poo#30595 is fixed.

#9 Updated by qmsu over 3 years ago

  • Status changed from Rejected to In Progress
  • Assignee set to qmsu

This is another issue other than poo#30595.

The disk is 100% used during upgrade, which lead to test failures, like:
https://openqa.suse.de/tests/1403771/file/install_and_reboot-df.txt
https://openqa.suse.de/tests/1412184/file/install_and_reboot-df.txt

The hdd image is 40GB, it should be large enough to perform upgrade, even in all-addons && all-patterns case.
This issue is only observed on ppc64le, not on any other arches: x86_64, s390x, aarch64

This is a suspect product bug, and we need collect disk usage info before upgrade to make sure there is enough disk space for upgrading.

According to test log of proxyscc_upgrade_sles12sp3+sdk_allpatterns, which used the same hdd image with media_upgrade_sles12sp3+sdk_allpatterns
https://openqa.suse.de/tests/1403713/file/upgrade_select-df.txt

There are 11GB free disk space before upgrade, with 70% disk used.
We need figure out two issues here:

  • why 11GB is not enough for upgrade to sle15, is it a product bug?
  • why 28GB (70% of 40GB) disk space is used before upgrade, is it a SLE 12-SP3 bug? who used the disk? should we try to clean up (i.e. snapshots) to release more disk space for upgrade?

#11 Updated by mitiao over 3 years ago

qmsu wrote:

This is another issue other than poo#30595.

The disk is 100% used during upgrade, which lead to test failures, like:
https://openqa.suse.de/tests/1403771/file/install_and_reboot-df.txt
https://openqa.suse.de/tests/1412184/file/install_and_reboot-df.txt

The hdd image is 40GB, it should be large enough to perform upgrade, even in all-addons && all-patterns case.
This issue is only observed on ppc64le, not on any other arches: x86_64, s390x, aarch64

This is a suspect product bug, and we need collect disk usage info before upgrade to make sure there is enough disk space for upgrading.

According to test log of proxyscc_upgrade_sles12sp3+sdk_allpatterns, which used the same hdd image with media_upgrade_sles12sp3+sdk_allpatterns
https://openqa.suse.de/tests/1403713/file/upgrade_select-df.txt

There are 11GB free disk space before upgrade, with 70% disk used.
We need figure out two issues here:

  • why 11GB is not enough for upgrade to sle15, is it a product bug?
  • why 28GB (70% of 40GB) disk space is used before upgrade, is it a SLE 12-SP3 bug? who used the disk? should we try to clean up (i.e. snapshots) to release more disk space for upgrade?

I remember that on ppc64le the snapshot size was much bigger than other archs during 12-SP3 phrase, the developer said we have to prepare more spaces for upgrade, but at that moment 40G was enough for upgrade.
Let's find out what happen of btrfs on ppc64le.

#12 Updated by qmsu over 3 years ago

  • Priority changed from Urgent to High

#13 Updated by qmsu over 3 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 40 to 100

Reported a bug for the disk issue on ppc64le:
[Bug 1079025] [ppc64le] btrfs consumed much more disk space compared to other architectures with similar rpm packages installed

Recreated hdd images with 80GB size for all-patterns tests:
SLES-12-SP3-ppc64le-allpatterns-updated.qcow2
SLES-12-SP3-ppc64le-ha+allpatterns-updated.qcow2
SLES-12-SP3-ppc64le-sdk+allpatterns-updated.qcow2
SLES-12-SP3-ppc64le-ha+sdk+allpatterns-updated.qcow2

Also available in: Atom PDF