Project

General

Profile

action #61094

o3: cleanup of snapshot-changes

Added by okurz 7 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Start date:
2019-12-17
Due date:
% Done:

100%

Estimated time:
Duration:

Description

Observation

ariel:/var/lib/openqa/share/factory # du -BM -d1
1M      ./tmp
377585M ./hdd
460845M ./iso
564647M ./repo
31157M  ./snapshot-changes
69747M  ./other
1503980M

so currently we have 31G for snapshot changes. We should have a cleanup strategy, asked DimStar in [#opensuse-factory](irc://chat.freenode.net/opensuse-factory)

History

#1 Updated by okurz 7 months ago

  • Status changed from New to Feedback
  • Assignee set to okurz
  • Target version set to Current Sprint
[17/12/2019 14:59:10] <DimStar> okurz: I'd say yes - certainly everything < 2019 is no longer needed (ever)
[17/12/2019 14:59:39] <DimStar> generally, it doesn't even make sense to keep more than 10 of the 'before the current' link ones
[17/12/2019 15:00:00] <DimStar> everything older is already as .txt on the download server
[17/12/2019 15:04:22] <okurz> cool, thx for the answer. So I will clean out manually once and we can then add a `find -mtime … -delete` in the scripts

did a manual rm $(ls | head -n -10).

https://gitlab.suse.de/openqa/scripts/merge_requests/425 for an automatic cleanup.

#2 Updated by okurz 6 months ago

this triggered the problem that newer snapshot changes were not recorded anymore. Reverted for now with https://gitlab.suse.de/openqa/scripts/merge_requests/433

Instead of calling rm $(ls … | head -n -10) from top level I should probably do that when the perl code decends into the individual directories.

#3 Updated by okurz 6 months ago

  • Assignee changed from okurz to andriinikitin

andriinikitin as you are currently working on covering o3 with ObsRsync I guess it makes sense if you also take this ticket to keep in mind that "snapshot-changes" need to be covered as well which is currently triggered within rsync.pl . The task for this ticket is to ensure that older changelog files are eventually pruned.

#4 Updated by andriinikitin 4 months ago

  • Status changed from Feedback to In Progress

#5 Updated by cdywan 3 months ago

  • Priority changed from Normal to Low

This can be considered low prio I think. The number is already down from the original report and most likely the changelogs won't make a huge difference:

cdywan@ariel:~> du -BM -d1 /var/lib/openqa/share/factory/snapshot-changes
2692M   /var/lib/openqa/share/factory/snapshot-changes

#6 Updated by andriinikitin 3 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

This should be covered by new cron job:

$ cat /etc/cron.d/factory-news-maintenance 
# remove files excluding 60 the newest for each arch and flavor
# weekly
0 1 * * SUN  find /var/lib/snapshot-changes/opensuse/Tumbleweed/         -maxdepth 1 -mindepth 1 -type f -printf '%T@\t%p\n' | sort -t $'\t' -g | head -n -60 | cut -d $'\t' -f 2- | xargs rm
15 1 * * SUN find /var/lib/snapshot-changes/opensuse-aarch64/Tumbleweed/ -maxdepth 1 -mindepth 1 -type f -printf '%T@\t%p\n' | sort -t $'\t' -g | head -n -60 | cut -d $'\t' -f 2- | xargs rm
30 1 * * SUN find /var/lib/snapshot-changes/opensuse/15.2/               -maxdepth 1 -mindepth 1 -type f -printf '%T@\t%p\n' | sort -t $'\t' -g | head -n -60 | cut -d $'\t' -f 2- | xargs rm
45 1 * * SUN find /var/lib/snapshot-changes/opensuse-aarch64/15.2/       -maxdepth 1 -mindepth 1 -type f -printf '%T@\t%p\n' | sort -t $'\t' -g | head -n -60 | cut -d $'\t' -f 2- | xargs rm
# monthly
0 2 1 * *    find /var/lib/snapshot-changes/microos/Tumbleweed/          -maxdepth 1 -mindepth 1 -type f -printf '%T@\t%p\n' | sort -t $'\t' -g | head -n -60 | cut -d $'\t' -f 2- | xargs rm
15 2 1 * *   find /var/lib/snapshot-changes/microos-aarch64/Tumbleweed/  -maxdepth 1 -mindepth 1 -type f -printf '%T@\t%p\n' | sort -t $'\t' -g | head -n -60 | cut -d $'\t' -f 2- | xargs rm
0 2 1 * *    find /var/lib/snapshot-changes/kubic/Tumbleweed/            -maxdepth 1 -mindepth 1 -type f -printf '%T@\t%p\n' | sort -t $'\t' -g | head -n -60 | cut -d $'\t' -f 2- | xargs rm
30 2 1 * *   find /var/lib/snapshot-changes/kubic-aarch64/Tumbleweed/    -maxdepth 1 -mindepth 1 -type f -printf '%T@\t%p\n' | sort -t $'\t' -g | head -n -60 | cut -d $'\t' -f 2- | xargs rm

Also available in: Atom PDF