action #30249
closed[sle][migration][sle15][ppc64le] test fails in install_and_reboot - failed by diskspace is exhausted
Added by JWSun over 6 years ago. Updated about 6 years ago.
100%
Description
Observation¶
openQA test in scenario sle-15-Installer-DVD-ppc64le-media_upgrade_sles12sp3+sdk_allpatterns@ppc64le fails in
install_and_reboot
Reproducible¶
Fails since (at least) Build 414.13
Expected result¶
Last good: (unknown) (or more recent)
Further details¶
Always latest result in this scenario: latest
Updated by qmsu over 6 years ago
- Subject changed from [sle][migration][sle15][ppc64le] test fails in install_and_reboot - failed by diskspace is exhausted to [sle][migration][sle15][ppc64le][medium] test fails in install_and_reboot - failed by diskspace is exhausted
Updated by qmsu over 6 years ago
- Subject changed from [sle][migration][sle15][ppc64le][medium] test fails in install_and_reboot - failed by diskspace is exhausted to [sle][migration][sle15][ppc64le] test fails in install_and_reboot - failed by diskspace is exhausted
Updated by JWSun over 6 years ago
More case failed by it
https://openqa.suse.de/tests/1399236
Updated by qmsu over 6 years ago
- Priority changed from High to Urgent
More tests are failed due to this issue.
Updated by okurz over 6 years ago
- Is duplicate of action #30595: [ppc64le] Deploy bcache on tmpfs workers added
Updated by qmsu over 6 years ago
Likely another issue besides poo#30595, because test log showed 40G disk space of VM was exhausted
https://openqa.suse.de/tests/1380164/file/install_and_reboot-df.txt
/dev/vda3 40384512 39743552 6272 100% /mnt
I will check again with new test runs when poo#30595 is fixed.
Updated by qmsu over 6 years ago
- Status changed from Rejected to In Progress
- Assignee set to qmsu
This is another issue other than poo#30595.
The disk is 100% used during upgrade, which lead to test failures, like:
https://openqa.suse.de/tests/1403771/file/install_and_reboot-df.txt
https://openqa.suse.de/tests/1412184/file/install_and_reboot-df.txt
The hdd image is 40GB, it should be large enough to perform upgrade, even in all-addons && all-patterns case.
This issue is only observed on ppc64le, not on any other arches: x86_64, s390x, aarch64
This is a suspect product bug, and we need collect disk usage info before upgrade to make sure there is enough disk space for upgrading.
According to test log of proxyscc_upgrade_sles12sp3+sdk_allpatterns, which used the same hdd image with media_upgrade_sles12sp3+sdk_allpatterns
https://openqa.suse.de/tests/1403713/file/upgrade_select-df.txt
There are 11GB free disk space before upgrade, with 70% disk used.
We need figure out two issues here:
- why 11GB is not enough for upgrade to sle15, is it a product bug?
- why 28GB (70% of 40GB) disk space is used before upgrade, is it a SLE 12-SP3 bug? who used the disk? should we try to clean up (i.e. snapshots) to release more disk space for upgrade?
Updated by qmsu over 6 years ago
- % Done changed from 0 to 40
Updated by mitiao over 6 years ago
qmsu wrote:
This is another issue other than poo#30595.
The disk is 100% used during upgrade, which lead to test failures, like:
https://openqa.suse.de/tests/1403771/file/install_and_reboot-df.txt
https://openqa.suse.de/tests/1412184/file/install_and_reboot-df.txtThe hdd image is 40GB, it should be large enough to perform upgrade, even in all-addons && all-patterns case.
This issue is only observed on ppc64le, not on any other arches: x86_64, s390x, aarch64This is a suspect product bug, and we need collect disk usage info before upgrade to make sure there is enough disk space for upgrading.
According to test log of proxyscc_upgrade_sles12sp3+sdk_allpatterns, which used the same hdd image with media_upgrade_sles12sp3+sdk_allpatterns
https://openqa.suse.de/tests/1403713/file/upgrade_select-df.txtThere are 11GB free disk space before upgrade, with 70% disk used.
We need figure out two issues here:
- why 11GB is not enough for upgrade to sle15, is it a product bug?
- why 28GB (70% of 40GB) disk space is used before upgrade, is it a SLE 12-SP3 bug? who used the disk? should we try to clean up (i.e. snapshots) to release more disk space for upgrade?
I remember that on ppc64le the snapshot size was much bigger than other archs during 12-SP3 phrase, the developer said we have to prepare more spaces for upgrade, but at that moment 40G was enough for upgrade.
Let's find out what happen of btrfs on ppc64le.
Updated by qmsu about 6 years ago
- Status changed from In Progress to Resolved
- % Done changed from 40 to 100
Reported a bug for the disk issue on ppc64le:
[Bug 1079025] [ppc64le] btrfs consumed much more disk space compared to other architectures with similar rpm packages installed
Recreated hdd images with 80GB size for all-patterns tests:
SLES-12-SP3-ppc64le-allpatterns-updated.qcow2
SLES-12-SP3-ppc64le-ha+allpatterns-updated.qcow2
SLES-12-SP3-ppc64le-sdk+allpatterns-updated.qcow2
SLES-12-SP3-ppc64le-ha+sdk+allpatterns-updated.qcow2