action #56453
closedAsset limit for parent group
Description
Right now we the asset limit of a parent group is just the default for the job groups. The problem we face is that more and more teams
split their work load into more and more job groups and the asset limiter has real trouble shuffling around the assets there.
I think it's time to set the limit on the parent group and count all job groups in it to one pool of disk space.
Updated by mkittler over 4 years ago
It would be a bit weird that the asset limit is applied on parent group level while the other cleanup related properties (storage durations for results and logs) are not. However, since the other limits are only time constraints it would not make much of a difference.
So yes, generally it makes sense to enforce the limit also on parent group level. But we still need to have the limit on job group level as well because some job groups might not have a parent.
Note that this would be a breaking change if we just implemented it without further precautions. If suddenly the limits set on the parent group level would be applied a lot of assets/results would be deleted because right now the value set on parent group level is only supposed to be the default for individual job groups within the parent. Not sure how to make the transition nicely. We could make the new behavior configurable but the logic for limiting assets is complicated enough so I don't like that idea. Maybe we can add a database migration which would set the limit of job groups explicitly which right now only inherit the limit from the parent. Then the migration would set the limit of the parent groups to the sum of the limit of their job groups. This way it wouldn't be a breaking change and only mess a little bit with the configuration. The defaults in the config file should possibly adjusted/increased, too.
The way we display this on the web UI (assets table and tree) would need to be adapted as well.
Updated by coolo over 4 years ago
I'm not saying it's easy to do - but we need to act. You're right that it needs to be consistent and it doesn't have to be all or nothing. I guess a parent job group setting can't be both default and catch all, so possibly we need a new setting that only applies to parent job groups and then the children's job group settings are grayed out if set. That it makes the limit_asset task more complicated is a given, but I don't think by much.
Updated by mkittler over 4 years ago
- Assignee set to mkittler
- Target version changed from Ready to Current Sprint
Updated by mkittler over 4 years ago
Draft: https://github.com/os-autoinst/openQA/pull/2536
The draft includes documentation changes and a description explaining how I'd implement that feature.
Updated by mkittler over 4 years ago
- Status changed from New to In Progress
This has already been in progress for a while now. The PR is ready to review: https://github.com/os-autoinst/openQA/pull/2536
Updated by mkittler over 4 years ago
- Status changed from In Progress to New
Setting back to 'New' because I'm not actively working on it anymore. That is because @okurz mentioned concerns in the PR and the PR received not any other feedback.
Note that the PR makes the limiting much more coarse, indeed. This is also a concern I mentioned myself in the chat. Hence I would have implemented the parent group asset limit as an additional constraint and not an overriding constraint. I made it an overriding constraint because @coolo suggested that in the chat. Unfortunately the rest of the team didn't really participate in the discussion. Unless we clarify what we want it makes no sense to continue working on this issue.
Updated by mkittler over 4 years ago
- Status changed from New to In Progress
We've been deciding to make this feature optional. To make not use of it one just has to not specify the size limit on parent group level. That's also the default. Size limits which were previously inherited from the parent limit are migrated to explicit size limits on job group level. The PR has already been updated and I'd merge it next year.
Updated by mkittler over 4 years ago
- Status changed from In Progress to Feedback
Deployed on o3 and it works now. However, on o3 we don't really (want) to use the new feature so let's wait how well it works on OSD.
Updated by mkittler over 4 years ago
- Status changed from Feedback to Resolved
- Target version changed from Current Sprint to Done
Deployed on OSD and used for the SLE 15 parent group. When browsing the relevant web UI pages it looks as expected (assets are accounted to the parent group; the editor for group properties works).