Project

General

Profile

action #59933

Prevent depletion of space on / with extra partition or quota on /home on o3 size:M

Added by okurz about 2 years ago. Updated 2 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Target version:
Start date:
2019-11-18
Due date:
2021-12-07
% Done:

0%

Estimated time:

Description

Observation

Happened again that we receive a monitoring alert for / being full (but recovered soon after) after a user logged in, did some actions involing bigger space needs and realized the problem.

Problem

This can cause disruption of services. The root volume is only 10GB for both osd and o3 which is ok for VMs but users do not realize that /home and /tmp is therefore not able to hold big data.

Acceptance criteria

  • AC1: /home on o3 is on separate storage
  • AC2: DONE same on osd

Suggestions

  • Enable user quotas (using aquota)
  • Mount another partition for /home, e.g. a qcow file, same as on OSD where we already have that

Related issues

Related to openQA Infrastructure - action #70939: [alert] ** PROBLEM Service Alert: ariel-opensuse.suse.de/root partition is WARNING **Resolved2020-09-04

History

#1 Updated by nicksinger about 2 years ago

I now created a (raw) disk on OSD located at /srv/homes.img (5G in size) and added an according fstab entry for it. I did go with a raw file since it is easier compared to a qcow2 to mount it via fstab. This should for now at least ease our problems with users writing too big files into their homes.

#2 Updated by okurz over 1 year ago

  • Priority changed from Normal to Low

#3 Updated by okurz over 1 year ago

  • Related to action #70939: [alert] ** PROBLEM Service Alert: ariel-opensuse.suse.de/root partition is WARNING ** added

#4 Updated by mkittler over 1 year ago

On o3 we still have 3 TiB disk space where we could place the home directories on: /dev/vdb1 5,0T 2,1T 3,0T 42% /space

So I guess we could easily move /home to that partition and bind-mount that directory as /home.

#5 Updated by okurz over 1 year ago

I suggest we follow #59933#note-1 with the same approach. This way we can limit the space that is used by /home

#6 Updated by okurz over 1 year ago

  • Target version set to Ready

#7 Updated by okurz about 1 year ago

  • Target version changed from Ready to future

#8 Updated by okurz 2 months ago

  • Due date set to 2021-12-07
  • Assignee set to nicksinger
  • Target version changed from future to Ready

nicksinger asked about this ticket. As stated in https://github.com/os-autoinst/scripts/blob/master/openqa-investigate#L59 if working in a related area right now we can add it to the backlog. As Nick stated "Should be an easy target to hit till end of the week", let's see how this goes then :)

#9 Updated by cdywan 2 months ago

  • Subject changed from Prevent depletion of space on / with extra partition or quota on /home and/or /tmp (both o3 and osd) to Prevent depletion of space on / with extra partition or quota on /home on o3 size:M
  • Description updated (diff)
  • Status changed from New to Workable

#10 Updated by nicksinger 2 months ago

  • Status changed from Workable to Resolved

I created a file-based hdd in /space/homes.img with 5G allocated. We currently only use ~400MB for all in /home so I assume 5G should be plenty and hopefully reminds people not to pollute their home directories too much but also can be enlarged anytime in the future.
Entry added to /etc/fstab for automatic configuration/mounting:

/space/homes.img /home ext4 defaults 1 1

and tested with mount -a. Old /home/* content moved to /space/OLD_HOMES_DELETEME_AFTER_2021_12_25_POO59933 because rm felt wrong and in case anything went wrong we can copy it back.
Quick relogin with my own account doesn't seem to show any problems (permissions all preserved, noting missing). Please reopen if you find any issues.

Also available in: Atom PDF