Project

General

Profile

Actions
Copy link

action #160481

closed

backup-vm: partitions usage (%) alert & systemd services alert size:S

Added by tinita 7 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
High
Assignee:
mkittler
Category:
Regressions/Crashes
Target version:
openQA Project (public) - Ready
Start date:
2024-05-17
Due date:
2024-06-06
% Done:

0%

Estimated time:
Tags:
alert, infra, reactive work

Description

Observation

Fri, 17 May 2024 04:01:33 +0200

1 firing alert instance
[IMAGE]

📁 GROUPED BY 

hostname=backup-vm

  🔥 1 firing instances

Firing [stats.openqa-monitor.qa.suse.de]
backup-vm: partitions usage (%) alert
View alert [stats.openqa-monitor.qa.suse.de]
Values
A0=86.0003690373683 
Labels
alertname
backup-vm: partitions usage (%) alert
grafana_folder

http://stats.openqa-monitor.qa.suse.de/alerting/grafana/partitions_usage_alert_backup-vm/view?orgId=1

Also, possibly related:
https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1
Not related, this was about backup-qam (and not backup-vm which this ticket is about):

Failed systemd services
2024-05-16 15:27:50    backup-qam    check-for-kernel-crash, kdump-notify

Suggestions

  • Check partition usage and which component contributes the most space usage
  • Check what happened that we had this short high usage surge
  • Consider increasing the size of the virtually attached storage
  • Consider tweaking our backup rules to either include less or less retention Not useful, it was the root partition (but backups are on the separate partition /dev/vdb1).
  • Or maybe don't do anything if this only happened once and is not likely to happen again based on monitoring data investigation
Actions
Copy link
 #1

Updated by tinita 7 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by tinita 7 months ago

Actions
Copy link
 #3

Updated by tinita 7 months ago

  • Subject changed from backup-vm: partitions usage (%) alert to backup-vm: partitions usage (%) alert & systemd services alert
  • Description updated (diff)
Actions
Copy link
 #4

Updated by okurz 7 months ago

  • Tags set to infra, alert, reactive work
Actions
Copy link
 #5

Updated by livdywan 7 months ago

  • Subject changed from backup-vm: partitions usage (%) alert & systemd services alert to backup-vm: partitions usage (%) alert & systemd services alert size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #6

Updated by livdywan 7 months ago

  • Description updated (diff)
Actions
Copy link
 #7

Updated by livdywan 7 months ago

  • Description updated (diff)
Actions
Copy link
 #8

Updated by mkittler 7 months ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions
Copy link
 #10

Updated by mkittler 7 months ago

  • Description updated (diff)
Actions
Copy link
 #11

Updated by mkittler 7 months ago

Looks like an update was going on at the time. Probably some snapper cleanup "fixed" the problem later. Considering the file system reached only 86.2 % there was no real problem. Maybe we should just bump the threshold to 90 % because always expecting so much headroom seems a bit wasteful.

I freed up almost 500 MiB by uninstalling libLLVM7 libLLVM9 libLLVM11 libLLVM15 webkit2gtk-4_0-injected-bundles webkit2gtk-4_1-injected-bundles WebKitGTK-4.0-lang gnome-online-accounts gnome-online-accounts-lang. 500 MiB is not that much of course. According to ncdu the root filesystem isn't really bloated so I don't think there's much more to gain here, though. According to snapper list there's also not one big snapshot.

Actions
Copy link
 #12

Updated by openqa_review 7 months ago

  • Due date set to 2024-06-06

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #13

Updated by mkittler 7 months ago

  • Status changed from In Progress to Resolved

I extended the root filesystem (and all other required underlying layers) by 5 GiB. Now we're at 58.7 % utilization which is good enough.

Actions
Copy link

Also available in: Atom PDF