action #164988
closed
coordination #92323: [saga][epic] Scale up: Fine-grained control over use and removal of results, assets, test data
coordination #179888: [epic] Creating, tracking, accounting "supporting jobs"
Better accounting for openqa-investigation jobs size:S
Added by okurz 8 months ago.
Updated about 4 hours ago.
Category:
Feature requests
Description
Motivation¶
#164979 alerted us about /results being nearly full. We found that groupless jobs are now the biggest offender linked to heavy jobs failing often triggering also heavy openqa-investigate jobs.
Acceptance criteria¶
- AC1: Big investigation jobs will not fill up our disk space; we would instead just keep less of them.
Suggestions¶
- Count investigation jobs towards the group of the original job
- Investigation jobs are groupless to avoid being considered for the result of the according group
- It is probably also not wanted by users; investigation jobs should not cause normal jobs to be stored less long but still be kept for a short time.
- The way the cleanup algorithm currently works makes this also hard to implement. It goes though jobs group by group and factoring in groupless jobs here without good relations in the database is not straight forward / efficient.
- Use a dedicated group for all investigation jobs
- Sounds most promising - just create a new group and schedule investigation jobs to be part of it.
- There is a caveat: Having all investigation jobs in one group does not solve the problem that investigation jobs for a particular scenario become very big. If we put everything in one group one scenario might cause other investigation jobs to be stored only very shortly.
- Copied from action #164979: [alert][grafana] File systems alert for WebUI /results size:S added
- Tags deleted (
alert, infra)
- Description updated (diff)
- Target version changed from Tools - Next to Ready
- Subject changed from Better accounting for openqa-investigation jobs to Better accounting for openqa-investigation jobs size:S
- Description updated (diff)
- Status changed from New to Workable
- Priority changed from Normal to Low
- Parent task set to #92323
- Target version changed from Ready to Tools - Next
- Target version changed from Tools - Next to Ready
- Status changed from Workable to In Progress
- Assignee set to ybonatakis
- Status changed from In Progress to Feedback
ybonatakis wrote in #note-11:
https://github.com/os-autoinst/scripts/pull/381
struggling with the test. submitted only the change in the investigation script
I did add a test but I decided to go with what works as I couldnt make the new test case to work modifying the host explicitly. I guess there is not a real request to O3, but it would be nice to understand why test breaking in different ways with my other attempts to inject host.
- Status changed from Feedback to Workable
As you were confused during the daily please review this and let us know where you need feedback or help.
- Status changed from Workable to Feedback
Yet another idea: How about a new "Investigation" parent group, which can have sub groups per original group. If there is a investigation subgroup defined for a group, investigation jobs go there, otherwise as a fallback they go into the main Investigation (sub) group.
E.g. OpenQA investigation jobs would go into "Investigation - openQA", "Development / Agama Devel" would go into "Investigation - Development - Agama Devel". Others go into "Investigation / Misc".
And then such individual investigation subgroups can be configured to keep results/logs for a shorter time.
This way we don't have to define an extra group for every group, just for the big ones.
I can't see another way of doing this automatically if we have such different cases where some groups create a lot of investigation jobs and others don't.
tinita wrote in #note-16:
Yet another idea: How about a new "Investigation" parent group, which can have sub groups per original group. If there is a investigation subgroup defined for a group, investigation jobs go there, otherwise as a fallback they go into the main Investigation (sub) group.
E.g. OpenQA investigation jobs would go into "Investigation - openQA", "Development / Agama Devel" would go into "Investigation - Development - Agama Devel". Others go into "Investigation / Misc".
And then such individual investigation subgroups can be configured to keep results/logs for a shorter time.
This way we don't have to define an extra group for every group, just for the big ones.
I can't see another way of doing this automatically if we have such different cases where some groups create a lot of investigation jobs and others don't.
I kinda liked the idea in the first read. But may not need to go that far. The main problem in concern is to not have investigation job consuming disk space with logs and results, right?
If we have different groups we have to keep track of their settings. I would say it is unnecessary and it doesnt give us much benefits at the end.
- Related to coordination #179221: [epic] Support keeping only jobs in database, remove all logs, assets, test results sooner added
- Status changed from Feedback to Resolved
Also on O3:
ariel:/home/ybonatakis # grep investigation_gid -rn /etc/openqa/openqa.ini
313:job_done_hook_failed = env from_email=o3-admins@suse.de scheme=http enable_force_result=true email_unreviewed=true investigation_gid=132 exclude_group_regex='(Development|Open Build Service|Others|Kernel).*/.*' /opt/os-autoinst-scripts/openqa-label-known-issues-and-investigate-hook
315:job_done_hook = env scheme=http enable_force_result=true email_unreviewed=true investigation_gid=132 exclude_group_regex='(Development|Open Build Service|Others|Kernel).*/.*' /opt/os-autoinst-scripts/openqa-label-known-issues-and-investigate-hook
and group settings adjust as:
Keep logs for 5 days
Keep important logs for 15 days
Keep results for 25 days
Keep important results for 0 (default-no change) days
please check if anything does not look as expected. I see already jobs running on OSD and as such I am going to resolve this ticket and mark the jobs with its issue number
- Parent task changed from #92323 to #179888
- Copied to action #179894: [spike][timeboxed:10h] Count assets+results of openqa-investigate jobs towards the originating group added
3h ago I restarted the openqa-webui and someone restarted openqa-gru 1.5h ago on O3. There are some jobs running in https://openqa.opensuse.org/group_overview/132 but I see investigation jobs which are not in the group. gonna keep tracking
Also available in: Atom
PDF