action #16496

[tools][sprint 201711.2] display current disk space consumption of job groups

Added by lnussel about 3 years ago. Updated almost 2 years ago.

Status:ResolvedStart date:06/02/2017
Priority:HighDue date:
Assignee:mkittler% Done:

0%

Category:Feature requests
Target version:Done
Difficulty:
Duration:

Description

There's a nice ui for editing the parameters of a job group
https://openqa.opensuse.org/admin/edit_parent_group/3

However, I cannot see how much space is currently consumed by which assets that are tracked as part of the job group. That information is hidden in the GRU logs.


Related issues

Related to openQA Project - action #28328: job was triggered trying to download HDD image but it's a... Rejected 24/11/2017

History

#1 Updated by lnussel about 3 years ago

and the total sum of all size limits would be interesting too I guess to prevent overbooking of the disk :)

#2 Updated by coolo about 3 years ago

It's not so easy. Job groups can share assets, but their size limits apply per group. Only if an asset is out of the limit for all groups, it's deleted. And you shouldn't even come close to overbook the disk with assets - as the test results and temporary asset uploads are on the same disk.

#3 Updated by coolo over 2 years ago

  • Assignee set to mkittler
  • Priority changed from Normal to High
  • Target version set to Ready

We need to work on figuring out what eats our disk space. The asset list we have e.g. on osd has hundreds of assets - and it's unclear when they were used the last by what job group.

Marius, I see this as part of the asset list - the job groups of course also contain job test results and images, but the majority of space is eaten by assets.

#4 Updated by szarate about 2 years ago

  • Subject changed from display current disk space consumption of job groups to [tools][sprint 201711.2] display current disk space consumption of job groups

#5 Updated by mkittler about 2 years ago

About the initial issue:

It would be grouping the assets by job group and show the accumulated size. And yes, by default this grouping would also include assets shared with other job groups. However, it would be possible to accumulate only exclusively used assets.

Not sure whether the grouping can be done with the help of the database (SQL GROUP BY statement) rather than doing it manually. When having to filter out shared assets it is likely only possible to do it manually.

But the variant to filter out shared assets makes likely most sense because only exclusive assets are definite candidates for cleanup (and cleanup is the purpose and only use for the size limit).


About making this part of the assets list:

and it's unclear when they were used the last by what job group

...

I see this as part of the asset list

So the table of assets should show the last use? And 'the last use' includes - beside the date - the job group which has the job which lastly referenced it?

I also noticed that we actually save the size of each asset in the database but do not display it in the asset table in the web UI. At least that could be easily improved.

#6 Updated by coolo about 2 years ago

You can also commit data back into the DB from the GRU tasks - that calculates a lot already anyway.

#7 Updated by okurz about 2 years ago

  • Related to action #28328: job was triggered trying to download HDD image but it's already gone added

#8 Updated by mkittler about 2 years ago

  • Status changed from New to In Progress

#9 Updated by szarate about 2 years ago

  • Target version changed from Ready to Current Sprint

Carry over to sprint 201712.1, assigning to "Current Sprint".

#10 Updated by mkittler about 2 years ago

  • Target version changed from Current Sprint to Ready

#11 Updated by mkittler about 2 years ago

  • Target version changed from Ready to Current Sprint

#12 Updated by mkittler about 2 years ago

  • Status changed from In Progress to Resolved

https://openqa.opensuse.org/admin/assets now shows the asset usage by group so the info is no longer hidden in the GRU logs and it works in production.

Note that there's also https://github.com/os-autoinst/openQA/pull/1553#issuecomment-356277916 which fixes the asset deletion from the web UI. Since this is not really requested here, I close this issue.

If anything is missing, just reopen.

#13 Updated by szarate almost 2 years ago

  • Target version changed from Current Sprint to Done

Also available in: Atom PDF