action #59008

900GB of assets not tracked ('other' assets?)

Added by coolo 4 months ago. Updated 3 months ago.

Status:ResolvedStart date:04/11/2019
Priority:HighDue date:
Assignee:cdywan% Done:

0%

Category:Concrete Bugs
Target version:Done
Difficulty:
Duration:

Description

/admin/assets knows about 3590GB, but our file system tracks 4400GB. There is /assets/tests with 10GB and there is /assets/factory/tmp with 1.3GB - the rest should be tracked. So where are the remaining 900GB?

I'm afraid it's LTP madness - other/ directory is at least ~900GB. And we have e.g. other/00678816-autoinst.xml from 2016. We may not scan this directory for untracked assets at all.


Related issues

Related to openQA Project - action #59103: Remove CURRENT repositories that are older than a month Resolved 05/11/2019
Copied to openQA Project - action #60713: left-over files in /var/lib/openqa/share/factory/tmp Resolved 04/11/2019

History

#1 Updated by coolo 4 months ago

  • Priority changed from Normal to Urgent

We have 72 assets in our DB that matches other/*-autoinst.xml, but have 6599 such files. As we get multiple times a day a notification about full disk, I set this to urgent

#2 Updated by cdywan 4 months ago

  • Assignee set to cdywan
  • Target version changed from Ready to Current Sprint

Taking a look into how/what is happening with those untracked files.

Leaving the status untouched as I'm evaluating what I'm looking at.

#3 Updated by cdywan 4 months ago

  • Related to action #59103: Remove CURRENT repositories that are older than a month added

#4 Updated by cdywan 4 months ago

  • Status changed from New to In Progress

As I'm studying the somewhat underdocumented code it's becoming apparent that #59103 and #57785 are part of this. More generally there are a lot of implicit corner cases which aren't documented and look very accidental.

#5 Updated by coolo 4 months ago

I did a one off, counting 'other' also as untracked assets and cleaning up those and now we're at 3547.72GiB tracked assets and 3828.09GiB used in /assets. That's still a significant delta, but closer to the truth.

#6 Updated by coolo 4 months ago

  • Priority changed from Urgent to High

#7 Updated by andriinikitin 4 months ago

coolo wrote:

3547.72GiB tracked assets and 3828.09GiB used in /assets.

It looks the number for tracked assets is based on apparent file size, while actual file size may be much bigger, depending on number of small files.

E.g. I've copied 15 files starting with 01* from factory/other and got difference in apparent vs actual file size 136Kb vs 184Kb :

> mkdir test
> cp /var/lib/openqa/factory/other/01* test/
> du -s --apparent test
136 test
> du -s test
184 test
> ls test/* | wc -l
15

Maybe we should calculate actual file size for assets as well (which may be tricky).

#8 Updated by okurz 3 months ago

I think you forgot to mention that you did proceed. There was e.g. https://github.com/os-autoinst/openQA/pull/2460 which is merged by now.

What we received from o3 this morning:

will be deleted in -359 days
…
[2019-11-15T05:35:29.0505 UTC] [warn] [pid:7212] Asset other/ltp-net_stress.ipsec_udp-opensuse-Tumbleweed-ppc64le-20190305-DVD@ppc64le-with-ltp.qcow2.txt is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0506 UTC] [warn] [pid:7212] Asset other/ltp-fcntl-locktests-Tumbleweed-20180707 is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0508 UTC] [warn] [pid:7212] Asset other/ltp-controllers-opensuse-Tumbleweed-aarch64-20191009-DVD@aarch64-with-ltp-qcow2.txt is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0510 UTC] [warn] [pid:7212] Asset other/00683337-autoinst.xml is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0512 UTC] [warn] [pid:7212] Asset other/ltp-net.features-opensuse-Tumbleweed-x86_64-20190110-DVD@64bit-with-ltp.qcow2.txt is not in any job group and will be deleted in -7929 days

EDIT: and they are so far not being deleted.

#10 Updated by okurz 3 months ago

  • Copied to action #60713: left-over files in /var/lib/openqa/share/factory/tmp added

#11 Updated by okurz 3 months ago

  • Status changed from In Progress to Resolved
  • Target version changed from Current Sprint to Done

osd:/var/lib/openqa/share/factory/other has currently 243G and it seems these are properly tracked as assets. OSD states /assets has 4391G used, https://openqa.suse.de/admin/assets tracks 4044G so we are "missing" 347G from the tracking. Created #60713 for /var/lib/openqa/share/factory/tmp which leaves 344G. Given that we have many small files and I do not know how "du" and "df" calculate used space differently this could be the difference depending on the filesystem inode size or metadata. I guess we are good.

Also available in: Atom PDF