action #59933
closedPrevent depletion of space on / with extra partition or quota on /home on o3 size:M
0%
Description
Observation¶
Happened again that we receive a monitoring alert for / being full (but recovered soon after) after a user logged in, did some actions involing bigger space needs and realized the problem.
Problem¶
This can cause disruption of services. The root volume is only 10GB for both osd and o3 which is ok for VMs but users do not realize that /home and /tmp is therefore not able to hold big data.
Acceptance criteria¶
- AC1: /home on o3 is on separate storage
- AC2: DONE same on osd
Suggestions¶
- Enable user quotas (using aquota)
- Mount another partition for /home, e.g. a qcow file, same as on OSD where we already have that
Updated by nicksinger over 4 years ago
I now created a (raw) disk on OSD located at /srv/homes.img (5G in size) and added an according fstab entry for it. I did go with a raw file since it is easier compared to a qcow2 to mount it via fstab. This should for now at least ease our problems with users writing too big files into their homes.
Updated by okurz over 3 years ago
- Related to action #70939: [alert] ** PROBLEM Service Alert: ariel-opensuse.suse.de/root partition is WARNING ** added
Updated by mkittler over 3 years ago
On o3 we still have 3 TiB disk space where we could place the home directories on: /dev/vdb1 5,0T 2,1T 3,0T 42% /space
So I guess we could easily move /home to that partition and bind-mount that directory as /home.
Updated by okurz over 3 years ago
I suggest we follow #59933#note-1 with the same approach. This way we can limit the space that is used by /home
Updated by okurz over 2 years ago
- Due date set to 2021-12-07
- Assignee set to nicksinger
- Target version changed from future to Ready
nicksinger asked about this ticket. As stated in https://github.com/os-autoinst/scripts/blob/master/openqa-investigate#L59 if working in a related area right now we can add it to the backlog. As Nick stated "Should be an easy target to hit till end of the week", let's see how this goes then :)
Updated by livdywan over 2 years ago
- Subject changed from Prevent depletion of space on / with extra partition or quota on /home and/or /tmp (both o3 and osd) to Prevent depletion of space on / with extra partition or quota on /home on o3 size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by nicksinger over 2 years ago
- Status changed from Workable to Resolved
I created a file-based hdd in /space/homes.img with 5G allocated. We currently only use ~400MB for all in /home so I assume 5G should be plenty and hopefully reminds people not to pollute their home directories too much but also can be enlarged anytime in the future.
Entry added to /etc/fstab for automatic configuration/mounting:
/space/homes.img /home ext4 defaults 1 1
and tested with mount -a. Old /home/* content moved to /space/OLD_HOMES_DELETEME_AFTER_2021_12_25_POO59933 because rm felt wrong and in case anything went wrong we can copy it back.
Quick relogin with my own account doesn't seem to show any problems (permissions all preserved, noting missing). Please reopen if you find any issues.