Project

General

Profile

Actions

action #59008

closed

900GB of assets not tracked ('other' assets?)

Added by coolo about 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2019-11-04
Due date:
% Done:

0%

Estimated time:

Description

/admin/assets knows about 3590GB, but our file system tracks 4400GB. There is /assets/tests with 10GB and there is /assets/factory/tmp with 1.3GB - the rest should be tracked. So where are the remaining 900GB?

I'm afraid it's LTP madness - other/ directory is at least ~900GB. And we have e.g. other/00678816-autoinst.xml from 2016. We may not scan this directory for untracked assets at all.


Related issues 2 (0 open2 closed)

Related to openQA Project (public) - action #59103: Remove CURRENT repositories that are older than a monthResolvedlivdywan2019-11-05

Actions
Copied to openQA Project (public) - action #60713: left-over files in /var/lib/openqa/share/factory/tmpResolvedmkittler2019-11-04

Actions
Actions #1

Updated by coolo about 5 years ago

  • Priority changed from Normal to Urgent

We have 72 assets in our DB that matches other/*-autoinst.xml, but have 6599 such files. As we get multiple times a day a notification about full disk, I set this to urgent

Actions #2

Updated by livdywan about 5 years ago

  • Assignee set to livdywan
  • Target version changed from Ready to Current Sprint

Taking a look into how/what is happening with those untracked files.

Leaving the status untouched as I'm evaluating what I'm looking at.

Actions #3

Updated by livdywan about 5 years ago

  • Related to action #59103: Remove CURRENT repositories that are older than a month added
Actions #4

Updated by livdywan about 5 years ago

  • Status changed from New to In Progress

As I'm studying the somewhat underdocumented code it's becoming apparent that #59103 and #57785 are part of this. More generally there are a lot of implicit corner cases which aren't documented and look very accidental.

Actions #5

Updated by coolo about 5 years ago

I did a one off, counting 'other' also as untracked assets and cleaning up those and now we're at 3547.72GiB tracked assets and 3828.09GiB used in /assets. That's still a significant delta, but closer to the truth.

Actions #6

Updated by coolo about 5 years ago

  • Priority changed from Urgent to High
Actions #7

Updated by andriinikitin about 5 years ago

coolo wrote:

3547.72GiB tracked assets and 3828.09GiB used in /assets.

It looks the number for tracked assets is based on apparent file size, while actual file size may be much bigger, depending on number of small files.

E.g. I've copied 15 files starting with 01* from factory/other and got difference in apparent vs actual file size 136Kb vs 184Kb :

> mkdir test
> cp /var/lib/openqa/factory/other/01* test/
> du -s --apparent test
136 test
> du -s test
184 test
> ls test/* | wc -l
15

Maybe we should calculate actual file size for assets as well (which may be tricky).

Actions #8

Updated by okurz about 5 years ago

I think you forgot to mention that you did proceed. There was e.g. https://github.com/os-autoinst/openQA/pull/2460 which is merged by now.

What we received from o3 this morning:

will be deleted in -359 days
…
[2019-11-15T05:35:29.0505 UTC] [warn] [pid:7212] Asset other/ltp-net_stress.ipsec_udp-opensuse-Tumbleweed-ppc64le-20190305-DVD@ppc64le-with-ltp.qcow2.txt is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0506 UTC] [warn] [pid:7212] Asset other/ltp-fcntl-locktests-Tumbleweed-20180707 is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0508 UTC] [warn] [pid:7212] Asset other/ltp-controllers-opensuse-Tumbleweed-aarch64-20191009-DVD@aarch64-with-ltp-qcow2.txt is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0510 UTC] [warn] [pid:7212] Asset other/00683337-autoinst.xml is not in any job group and will be deleted in -7929 days
[2019-11-15T05:35:29.0512 UTC] [warn] [pid:7212] Asset other/ltp-net.features-opensuse-Tumbleweed-x86_64-20190110-DVD@64bit-with-ltp.qcow2.txt is not in any job group and will be deleted in -7929 days

EDIT: and they are so far not being deleted.

Actions #10

Updated by okurz about 5 years ago

  • Copied to action #60713: left-over files in /var/lib/openqa/share/factory/tmp added
Actions #11

Updated by okurz about 5 years ago

  • Status changed from In Progress to Resolved
  • Target version changed from Current Sprint to Done

osd:/var/lib/openqa/share/factory/other has currently 243G and it seems these are properly tracked as assets. OSD states /assets has 4391G used, https://openqa.suse.de/admin/assets tracks 4044G so we are "missing" 347G from the tracking. Created #60713 for /var/lib/openqa/share/factory/tmp which leaves 344G. Given that we have many small files and I do not know how "du" and "df" calculate used space differently this could be the difference depending on the filesystem inode size or metadata. I guess we are good.

Actions

Also available in: Atom PDF