action #97976
[alert] OSD file systems - assets
50%
Description
Observation¶
https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?tab=alert&viewPanel=74&orgId=1&from=now-24h&to=now shows a constant increase since last midnight and there was no asset cleanup. https://openqa.suse.de/minion/locks shows currently "limit_tasks", "limit_screenshots_task", "limit_results_and_logs_task", "process_job_results_task" all to expire in 10h, and "limit_results_and_logs" currently running since 14 hours. Maybe that blocks (again?) asset cleanup?
Subtasks
Related issues
History
Updated by #2
Updated by okurz 11 months ago
https://openqa.suse.de/minion/jobs?id=2776306 says it was created 14 hours ago and is "inactive", why is that? And what does the runtime "a few seconds delay" mean?
#3
Updated by okurz 11 months ago
unlocked "limit_results_and_logs_task", should this block the assets cleanup? Now https://openqa.suse.de/minion/jobs?id=2776306 started. Attaching to the according gru process with strace -f -y … I see a lot of lookup of /var/lib/openqa/share/factory/other. This can take some time as the amount of files there seems to increase.
Monitoring progress on OSD.
#6
Updated by okurz 11 months ago
- Related to action #96789: File systems alert 90.256 assets used size:M added
#8
Updated by kraih 11 months ago
Updated by okurz wrote:
https://openqa.suse.de/minion/jobs?id=2776306 says it was created 14 hours ago and is "inactive", why is that? And what does the runtime "a few seconds delay" mean?
In case you didn't get an answer somewhere else yet. It means the job was enqueued or retried (retried in this case) with a delay of a few seconds. Probably because it checks the lock, and if it exists retries itself with the delay, over and over until it can get the lock itself.