Project

General

Profile

Actions
Copy link

action #71575

closed

[osd][alert] limited /assets - idea: ask EngInfra for slow+cheap storage from central server for /assets/fixed only

Added by okurz about 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
mkittler
Category:
-
Target version:
openQA Project (public) - Ready
Start date:
2020-09-02
Due date:
% Done:

0%

Estimated time:

Description

Motivation

/assets on osd is recently quite full. E.g. see #70885 . There are some ideas how to use storage more efficiently and also use our own "storage server". What might be a good intermediate solution (in parallel) would be to ask EngInfra for slow+cheap storage from central server for /assets/fixed only as the fixed assets are likely used by less jobs and we can live with a bit less performance.

Suggestions

  • Create ticket to ask EngInfra for slow+cheap storage, e.g. on netapp next to existing fast+expensive storage that we currently use (AFAIK)
  • Use new storage for specific mount point on /var/lib/openqa/share/factory/
  • Based on
/assets> find . -maxdepth 3 -type d -name 'fixed'
./factory/hdd/fixed
./factory/iso/fixed
./factory/repo/fixed

use the new storage for each of the above

Related issues 4 (0 open — 4 closed)

Related to openQA Infrastructure (public) - action #69667: missing monitoring data for vde after partitions where reorderedResolvedmkittler2020-08-06

Actions
Related to openQA Infrastructure (public) - action #119215: [openQA][repos][aarch64][15-SP5[Full Media] 15-SP5 Full media is missing from http://openqa.suse.de/assets/repo/Resolvedokurz2022-10-24

Actions
Copied from openQA Infrastructure (public) - action #70885: [osd][alert] flaky file system alert: /assetsResolvedokurz2020-09-022021-07-23

Actions
Copied to openQA Infrastructure (public) - action #73165: [osd] Consolidate "expensive+fast" and "cheap+slow" storage after realizing vdc is "cheap+slow" as wellResolvedokurz2020-09-02

Actions
Actions
Copy link
 #1

Updated by okurz about 4 years ago

  • Copied from action #70885: [osd][alert] flaky file system alert: /assets added
Actions
Copy link
 #2

Updated by okurz about 4 years ago

  • Assignee set to mkittler

@mkittler as discussed, please create an EngInfra ticket in the normal "infra" queue, put osd-admins@suse.de in CC, put "[openqa]" in the subject line, ask for slow+cheap (rotating disk) storage to our osd vm as additional partition or device. I would suggest something like at least 2TB if that is feasible. More is also useful.

Actions
Copy link
 #4

Updated by mkittler about 4 years ago

The new disk is there (/dev/vde). It has 2 TiB. I can copy over fixed assets of the 3 assets types and bind-mount the directories.

Actions
Copy link
 #5

Updated by okurz about 4 years ago

  • Related to action #69667: missing monitoring data for vde after partitions where reordered added
Actions
Copy link
 #6

Updated by mkittler about 4 years ago

  • Status changed from Workable to Feedback
  • Formatted /dev/vde as xfs in accordance with the regular assets partition and mounted as /space-slow.
  • Copied fixed isos over: rsync -aHP /assets/factory/iso/fixed/ /space-slow/assets/factory/iso/fixed/
  • Made use of the fixed asset at their new location¹.
  • Updated /etc/fstab.

It seems to work. If no problems come up we can delete /assets/bak and move the other fixed asset directories similarly.

¹

martchus@openqa:~> cat /home/martchus/mount-assets.sh 
#!/bin/sh
set -ex
rsync -aHP /assets/factory/iso/fixed/ /space-slow/assets/factory/iso/fixed/
systemctl stop openqa-webui openqa-gru
rsync -aHP /assets/factory/iso/fixed/ /space-slow/assets/factory/iso/fixed/
mv /assets/factory/iso/fixed /assets/bak
mkdir /assets/factory/iso/fixed
mount --bind /space-slow/assets/factory/iso/fixed /assets/factory/iso/fixed
systemctl start openqa-webui openqa-gru
Actions
Copy link
 #7

Updated by mkittler about 4 years ago

  • Status changed from Feedback to In Progress

I've removed /assets/bak and I'm continuing moving fixed HDDs and repos to the new storage.

Actions
Copy link
 #8

Updated by okurz about 4 years ago

I will extend the quota for SLE15 on osd a bit and inform sebchlad et al: "We moved some "fixed" assets to a separate storage sink so we have a bit more space to spare for assets. I increased the size limit for assets of the SLE15 parent job group on openqa.suse.de from 980GB to 1200GB. Still, any optimization on the side of HA images would be appreciated as well as the others ideas we suggested". Also SAP+HA team has created https://infra.nue.suse.com/SelfService/Display.html?id=178140 to ask for +2TB for /assets. I commented in the ticket that as alternative cheap+slow can be increased in size partially.

Actions
Copy link
 #9

Updated by mkittler about 4 years ago

I moved fixed HDDs as well and will remove /assets/bak (currently still holding 1.4 TB fixed HDDs) soon. For now I'll keep the repos where they are because /assets is at 60 % (when subtracting /assets/bak) and the new slow storage already at 79 %.

Actions
Copy link
 #10

Updated by mkittler about 4 years ago

I've been deleting the backups. With

/dev/vdc        5,0T    3,2T  1,9T   63% /assets
/dev/vdd        5,0T    3,7T  1,4T   73% /results
/dev/vde        2,0T    1,6T  441G   79% /space-slow

we have the partitions kind of balanced out so I'm leaving it like this. I'll extend our file systems alert to cover /space-slow.

Actions
Copy link
 #12

Updated by mkittler about 4 years ago

  • Status changed from In Progress to Resolved

Merged identical SR from @okurz for the alert.

Actions
Copy link
 #13

Updated by okurz about 4 years ago

  • Copied to action #73165: [osd] Consolidate "expensive+fast" and "cheap+slow" storage after realizing vdc is "cheap+slow" as well added
Actions
Copy link
 #14

Updated by okurz about 2 years ago

  • Related to action #119215: [openQA][repos][aarch64][15-SP5[Full Media] 15-SP5 Full media is missing from http://openqa.suse.de/assets/repo/ added
Actions
Copy link

Also available in: Atom PDF