action #173947
closed[alert] s390zl12: partitions usage (%) alert Generic partitions_usage_alert_s390zl12 generic size:S
0%
Description
Date: Mon, 09 Dec 2024 03:41:27 +0100
https://monitor.qa.suse.de/alerting/grafana/de4e513999b0b67eaa549ebfb7adb270d1735cf9/view?orgId=1
hostname=s390zl12
Suggestions¶
- Take a look at the partition dasda2 on s390zl12, e.g. check with ncdu/gdu for what's using the space.
- Cleanup some non-important files e.g. old log files, maybe adding automatic cleanup (judging by https://monitor.qa.suse.de/d/GDs390zl12/dashboard-for-s390zl12?viewPanel=panel-65090&from=now-90d&to=now&timezone=browser&var-datasource=000000001&refresh=1m there is no automatic cleanup but maybe some files have been deleted manually in the past)
Updated by okurz 3 months ago
- Related to action #170122: [alert][FIRING:1] s390zl13 (s390zl13: partitions usage (%) alert Generic partitions_usage_alert_s390zl13 generic) added
Updated by tinita 3 months ago
Looking at the last 90 days, the used disk space just goes up and up. there seems to be some cleanup happening, resolving the alert for a while.
https://monitor.qa.suse.de/d/GDs390zl12/dashboard-for-s390zl12?viewPanel=panel-65090&from=now-90d&to=now&timezone=browser&var-datasource=000000001&refresh=1m
Updated by mkittler 3 months ago
- Subject changed from [alert] s390zl12: partitions usage (%) alert Generic partitions_usage_alert_s390zl12 generic to [alert] s390zl12: partitions usage (%) alert Generic partitions_usage_alert_s390zl12 generic size:S
- Description updated (diff)
- Status changed from New to Workable
Updated by mkittler 3 months ago
- Status changed from In Progress to Resolved
There's not much gained from cleaning up further data in home directories but snapshots take a considerable amount of disk space:
martchus@s390zl12:~> sudo btrfs filesystem du -s /
Total Exclusive Set shared Filename
45.29GiB 6.95GiB 5.49GiB /
martchus@s390zl12:~> sudo btrfs filesystem du -s /home
Total Exclusive Set shared Filename
88.00KiB 88.00KiB 0.00B /home
martchus@s390zl12:~> sudo btrfs filesystem du -s /usr
Total Exclusive Set shared Filename
3.86GiB 0.00B 3.86GiB /usr
martchus@s390zl12:~> sudo btrfs filesystem du -s /.snapshots
Total Exclusive Set shared Filename
40.11GiB 1.76GiB 5.49GiB /.snapshots
Although it doesn't look as bad according to snapper:
martchus@s390zl12:~> sudo snapper list
# | Type | Pre # | Date | User | Used Space | Cleanup | Description | Userdata
-----+--------+-------+----------------------------------+------+------------+---------+-----------------------+--------------
0 | single | | | root | | | current |
400* | single | | Wed 08 May 2024 10:53:45 PM CEST | root | 29.92 MiB | | writable copy of #391 |
568 | pre | | Fri 15 Nov 2024 03:32:53 AM CET | root | 1.16 GiB | number | zypp(zypper) | important=yes
569 | post | 568 | Fri 15 Nov 2024 03:37:52 AM CET | root | 7.55 MiB | number | | important=yes
570 | pre | | Sun 17 Nov 2024 03:31:46 AM CET | root | 2.69 MiB | number | zypp(zypper) | important=yes
571 | post | 570 | Sun 17 Nov 2024 03:32:07 AM CET | root | 205.95 MiB | number | | important=yes
594 | pre | | Mon 09 Dec 2024 03:32:55 AM CET | root | 469.26 MiB | number | zypp(zypper) | important=no
595 | post | 594 | Mon 09 Dec 2024 03:33:08 AM CET | root | 2.02 MiB | number | | important=no
596 | pre | | Tue 10 Dec 2024 03:37:26 PM CET | root | 352.00 KiB | number | zypp(zypper) | important=no
597 | post | 596 | Tue 10 Dec 2024 03:37:28 PM CET | root | 416.00 KiB | number | | important=no
I also checked all the other subvolumes but none contain a significant amount of data. So maybe the snapshots do use a lot of space but snapper doesn't show it.
Considering we're below 80 % I'll leave it at that, though. Although it would be interesting if someone could shed some light on the snapshotting situation.
Updated by ybonatakis 18 days ago ยท Edited
- Status changed from Resolved to Workable
reopen it as there was a new alert today http://monitor.qa.suse.de/goto/hzIQSi5Hg?orgId=1
Updated by ybonatakis 18 days ago
but fixed two hours later. not sure if someone did anything