asset cleanup jobs do not run on o3 (results cleanup works), workaround: unlock locks manually
I am receiving monitoring alerts for o3 for usage of /space . https://thruk.suse.de/thruk/cgi-bin/extinfo.cgi?type=2&host=ariel-opensuse.suse.de&service=space%20partition#pnp_th2/1568826977/1568916977/0 shows the details. This happened on 2019-10-03 for the third time. I monitored the space usage and saw that /space was nearly depleted (100 GB free space left).
https://openqa.opensuse.org/minion/jobs?state=inactive I see "limit_assets". https://openqa.opensuse.org/minion/jobs?state=finished&offset=0&task=limit_assets do not show any successfully finished "limit_assets" jobs, https://openqa.opensuse.org/minion/jobs?state=finished&offset=0&task=limit_results_and_logs is fine though. https://openqa.opensuse.org/minion/jobs?state=failed mentions "limit_assets".
- Subject changed from asset and results cleanup jobs do not run on o3, workaround: unlock locks manually to asset cleanup jobs do not run on o3 (results cleanup works), workaround: unlock locks manually
- Description updated (diff)
- Priority changed from High to Normal
Wildly clicking around on https://openqa.opensuse.org/minion/jobs does something, e.g. restarting limit_assets jobs, removing the lock, removing failed or inactive. It seems a "limit_assets" job is running now. In strace I can see that the gru job iterates some repo folders. Why it needs to list every single file in an individual repo folder I do not know, seems inefficient. However it finished successfully for now.
- Due date set to 10/12/2019
- Status changed from New to Feedback
- Assignee set to okurz
we closed the PR meanwhile and wait for an upstream patch including https://github.com/mojolicious/minion/compare/d045708eeeb7...105612d4787b .
here is what I did on o3 for now:
$ cat /etc/systemd/system/openqa-gru.service.d/override.conf [Service] ExecStop=/usr/bin/psql openqa -c 'delete from minion_locks;'
waiting for https://build.opensuse.org/request/show/749518
EDIT: 2019-11-21: And https://github.com/os-autoinst/openQA/pull/2532 updated the dependencies for perl-Minion