Actions
action #138518
closedunreal6 partition usage alert
Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2023-10-25
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Actions
Added by livdywan over 1 year ago. Updated over 1 year ago.
0%
Description
See #131546 as well. There is a rule in our salt to add a Cron job cleaning up assets. Maybe we can use the second SSD in the machine or have it replaced with a bigger one, there are still some in the shelf
It seems the silence is not effective or was never created
discussing with mloviska about paths to improvement. I am thinking of replacing the second disk with a larger one.
The machine has two physical storage slots so I can not simply add a third. I will power down the machine and replace the one holding the VG with a 800GB-1TB one. Then we can also check network connectivity afterwards again.
unreal6:~ # lsblk -o NAME,SERIAL
NAME SERIAL
sda CVDA410106EZ1207GN
└─sda1
└─openqa_vg-openqa_lv
sdb CVDA410105VG1207GN
├─sdb1
├─sdb2
└─sdb3
so I will need to remove CVDA410106EZ1207GN and replace with a bigger one
Swapped disks and added 960GB as second. System booted into emergency mode (expected), in there disabled mount point relying on not anymore existing VG, continued boot and then
pvcreate /dev/sdb
vgcreate openqa_vg /dev/sdb
lvcreate -l 100%VG -n openqa_lv openqa_vg
mkfs.ext4 /dev/openqa_vg/openqa_lv
# enable back entry in /etc/fstab
mount -a
systemctl start libvirtd
monitoring jobs from https://openqa.suse.de/admin/workers referencing "unreal6" and retriggered according failures, e.g. https://openqa.suse.de/tests/12685089 scheduled just now, https://openqa.suse.de/tests/12684276#live running right now
Setting due date based on mean cycle time of SUSE QE Tools
A lot of tests passed using unreal6, e.g. https://openqa.suse.de/tests/12693711 and https://openqa.suse.de/tests/12684276 and https://openqa.suse.de/tests/12685122 . Alert is ok again. I have observed that the partition usage graphs are broken, reported #138650. Removed the silence accordingly. I will monitor over the next time.
No related alert, monitoring graphs still broken.
/dev/mapper/openqa_vg-openqa_lv 880G 84G 751G 11% /var/lib/libvirt/images
has more than enough free space.