Added by coolo over 4 years ago. Updated over 4 years ago.
Description
/admin/assets knows about 3590GB, but our file system tracks 4400GB. There is /assets/tests with 10GB and there is /assets/factory/tmp with 1.3GB - the rest should be tracked. So where are the remaining 900GB?
I'm afraid it's LTP madness - other/ directory is at least ~900GB. And we have e.g. other/00678816-autoinst.xml from 2016. We may not scan this directory for untracked assets at all.
We have 72 assets in our DB that matches other/*-autoinst.xml, but have 6599 such files. As we get multiple times a day a notification about full disk, I set this to urgent
Taking a look into how/what is happening with those untracked files.
Leaving the status untouched as I'm evaluating what I'm looking at.
I did a one off, counting 'other' also as untracked assets and cleaning up those and now we're at 3547.72GiB tracked assets and 3828.09GiB used in /assets. That's still a significant delta, but closer to the truth.
coolo wrote:
3547.72GiB tracked assets and 3828.09GiB used in /assets.
It looks the number for tracked assets is based on apparent file size, while actual file size may be much bigger, depending on number of small files.
E.g. I've copied 15 files starting with 01* from factory/other and got difference in apparent vs actual file size 136Kb vs 184Kb :
> mkdir test
> cp /var/lib/openqa/factory/other/01* test/
> du -s --apparent test
136 test
> du -s test
184 test
> ls test/* | wc -l
15
Maybe we should calculate actual file size for assets as well (which may be tricky).
I think you forgot to mention that you did proceed. There was e.g. https://github.com/os-autoinst/openQA/pull/2460 which is merged by now.
What we received from o3 this morning:
will be deleted in -359 days
…
[2019-11-15T05:35:29.0505 UTC] [warn] [pid:7212] Asset other/ltp-net_stress.ipsec_udp-opensuse-Tumbleweed-ppc64le-20190305-DVD@ppc64le-with-ltp.qcow2.txt is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0506 UTC] [warn] [pid:7212] Asset other/ltp-fcntl-locktests-Tumbleweed-20180707 is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0508 UTC] [warn] [pid:7212] Asset other/ltp-controllers-opensuse-Tumbleweed-aarch64-20191009-DVD@aarch64-with-ltp-qcow2.txt is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0510 UTC] [warn] [pid:7212] Asset other/00683337-autoinst.xml is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0512 UTC] [warn] [pid:7212] Asset other/ltp-net.features-opensuse-Tumbleweed-x86_64-20190110-DVD@64bit-with-ltp.qcow2.txt is not in any job group and will be deleted in -7929 days
EDIT: and they are so far not being deleted.
osd:/var/lib/openqa/share/factory/other has currently 243G and it seems these are properly tracked as assets. OSD states /assets has 4391G used, https://openqa.suse.de/admin/assets tracks 4044G so we are "missing" 347G from the tracking. Created #60713 for /var/lib/openqa/share/factory/tmp which leaves 344G. Given that we have many small files and I do not know how "du" and "df" calculate used space differently this could be the difference depending on the filesystem inode size or metadata. I guess we are good.