action #175791
opencoordination #161414: [epic] Improved salt based infrastructure management
[alert] storage: partitions usage (%) alert size:S
0%
Description
Observation¶
Values
A0=85.08932639307272
Labels
alertname storage: partitions usage (%) alert
grafana_folder Generic
hostname storage
rule_uid partitions_usage_alert_storage
type generic
So sda on the host storage is too full (85 % full).
http://monitor.qa.suse.de/d/GDstorage?orgId=1&viewPanel=65090
Suggestions¶
- Clean up storage, probably taken by backup of backup VM (see related ticket)
- Do not adjust the alert itself, it is perfectly fine
Updated by jbaier_cz 12 days ago
- Copied from action #150887: [alert] [FIRING:1] s390zl12 (s390zl12: partitions usage (%) alert Generic partitions_usage_alert_s390zl12 generic), also s390zl13 size:M added
Updated by okurz 12 days ago
- Related to action #173347: Ensure we have a current backup of qamaster VMs, VM config, jenkins data, data from backup-vm itself, etc. size:S added
Updated by gpathak 11 days ago · Edited
@okurz
I am planning to delete /storage/backup/backup-vm/ since this is duplicate of /storage/rsnapshot/
/storage/rsnapshot/ is always a latest up to date backup, I have to update the https://gitlab.suse.de/suse/wiki/-/blob/main/qe_infrastructure.md#backup-of-additional-services-running-on-qamaster accordingly if we choose to delete /storage/backup/backup-vm/
What are your thoughts? Can we move /storage/backup/backup-vm/ to some other machine?
Updated by gpathak 11 days ago
Cleaned-up /storage/backup/backup-vm/ and created MR https://gitlab.suse.de/suse/wiki/-/merge_requests/8/diffs
Updated by livdywan 11 days ago
- Status changed from Feedback to Resolved
gpathak wrote in #note-13:
Cleaned-up
/storage/backup/backup-vm/and created MR https://gitlab.suse.de/suse/wiki/-/merge_requests/8/diffs
Please remember an Urgent ticket should not remain in Feedback. If I see this correct it should be fixed, so let's resolve and re-open if there is any issues.
Updated by okurz 10 days ago
I think I misunderstood your proposal to delete backup-vm/ . I assumed you had an additional copy of backup-data.qcow2. Deleting backup-vm/ is in conflict with #173347. I suggest to bring back backup-vm/ and find more space elsewhere by either removing other data or ordering additional storage hardware.
Updated by gpathak 9 days ago
okurz wrote in #note-19:
I think I misunderstood your proposal to delete backup-vm/ . I assumed you had an additional copy of backup-data.qcow2. Deleting backup-vm/ is in conflict with #173347. I suggest to bring back backup-vm/ and find more space elsewhere by either removing other data or ordering additional storage hardware.
We cannot delete anything more from storage. Bringing back backup-vm/ will cause grafana alert to trigger again, we need to silence the alert until we have additional storage.
Updated by openqa_review 9 days ago
- Due date set to 2025-02-06
Setting due date based on mean cycle time of SUSE QE Tools