action #59008
closed900GB of assets not tracked ('other' assets?)
Description
/admin/assets knows about 3590GB, but our file system tracks 4400GB. There is /assets/tests with 10GB and there is /assets/factory/tmp with 1.3GB - the rest should be tracked. So where are the remaining 900GB?
I'm afraid it's LTP madness - other/ directory is at least ~900GB. And we have e.g. other/00678816-autoinst.xml from 2016. We may not scan this directory for untracked assets at all.
Updated by coolo over 4 years ago
- Priority changed from Normal to Urgent
We have 72 assets in our DB that matches other/*-autoinst.xml, but have 6599 such files. As we get multiple times a day a notification about full disk, I set this to urgent
Updated by livdywan over 4 years ago
- Assignee set to livdywan
- Target version changed from Ready to Current Sprint
Taking a look into how/what is happening with those untracked files.
Leaving the status untouched as I'm evaluating what I'm looking at.
Updated by livdywan over 4 years ago
- Related to action #59103: Remove CURRENT repositories that are older than a month added
Updated by livdywan over 4 years ago
- Status changed from New to In Progress
Updated by coolo over 4 years ago
I did a one off, counting 'other' also as untracked assets and cleaning up those and now we're at 3547.72GiB tracked assets and 3828.09GiB used in /assets. That's still a significant delta, but closer to the truth.
Updated by andriinikitin over 4 years ago
coolo wrote:
3547.72GiB tracked assets and 3828.09GiB used in /assets.
It looks the number for tracked assets is based on apparent file size, while actual file size may be much bigger, depending on number of small files.
E.g. I've copied 15 files starting with 01* from factory/other and got difference in apparent vs actual file size 136Kb vs 184Kb :
> mkdir test
> cp /var/lib/openqa/factory/other/01* test/
> du -s --apparent test
136 test
> du -s test
184 test
> ls test/* | wc -l
15
Maybe we should calculate actual file size for assets as well (which may be tricky).
Updated by okurz over 4 years ago
I think you forgot to mention that you did proceed. There was e.g. https://github.com/os-autoinst/openQA/pull/2460 which is merged by now.
What we received from o3 this morning:
will be deleted in -359 days
…
[2019-11-15T05:35:29.0505 UTC] [warn] [pid:7212] Asset other/ltp-net_stress.ipsec_udp-opensuse-Tumbleweed-ppc64le-20190305-DVD@ppc64le-with-ltp.qcow2.txt is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0506 UTC] [warn] [pid:7212] Asset other/ltp-fcntl-locktests-Tumbleweed-20180707 is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0508 UTC] [warn] [pid:7212] Asset other/ltp-controllers-opensuse-Tumbleweed-aarch64-20191009-DVD@aarch64-with-ltp-qcow2.txt is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0510 UTC] [warn] [pid:7212] Asset other/00683337-autoinst.xml is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0512 UTC] [warn] [pid:7212] Asset other/ltp-net.features-opensuse-Tumbleweed-x86_64-20190110-DVD@64bit-with-ltp.qcow2.txt is not in any job group and will be deleted in -7929 days
EDIT: and they are so far not being deleted.
Updated by livdywan over 4 years ago
Updated by okurz over 4 years ago
- Copied to action #60713: left-over files in /var/lib/openqa/share/factory/tmp added
Updated by okurz over 4 years ago
- Status changed from In Progress to Resolved
- Target version changed from Current Sprint to Done
osd:/var/lib/openqa/share/factory/other has currently 243G and it seems these are properly tracked as assets. OSD states /assets has 4391G used, https://openqa.suse.de/admin/assets tracks 4044G so we are "missing" 347G from the tracking. Created #60713 for /var/lib/openqa/share/factory/tmp which leaves 344G. Given that we have many small files and I do not know how "du" and "df" calculate used space differently this could be the difference depending on the filesystem inode size or metadata. I guess we are good.