action #164979
Updated by livdywan 7 days ago
## Observation Observed at 2024-08-06 07:17:00 +0200 CEST ~~One of the file systems~~ /results is too full (> 90%) See http://stats.openqa-monitor.qa.suse.de/d/WebuiDb?orgId=1&viewPanel=74 Current usage: ``` Filesystem Size Used Avail Use% Mounted on /dev/vdd 7.0T 6.4T 681G 91% /results ``` ## Acceptance criteria **AC1:** There is enough space and headroom on the affected file system /results, i.e. considerably more than 20% ## Suggestions * Check job group "logs" retention settings for "not-important" / "groupless" result and consider reducing the period * Consider extending the silence period if fixing takes too long: https://stats.openqa-monitor.qa.suse.de/alerting/silence/9ee9b299-3d06-4234-97bf-6b84e2ad9a24/edit?alertmanager=grafana * Reconsider the design of scheduling openqa-investigate for unreviewed jobs and possibly plan in a separate ticket * Tell the security squad that their test scenario(s) are problematic and should fail less or be properly reviewed * Tell the security squad about their test scenario(s) which is significantly bigger than other jobs and consider reducing the space usage, e.g. save less or compress stuff ## Rollback steps * **DONE** ~~Remove Remove silence https://stats.openqa-monitor.qa.suse.de/alerting/silence/9ee9b299-3d06-4234-97bf-6b84e2ad9a24/edit?alertmanager=grafana~~ https://stats.openqa-monitor.qa.suse.de/alerting/silence/9ee9b299-3d06-4234-97bf-6b84e2ad9a24/edit?alertmanager=grafana ## Out of scope * Better accounting e.g. linking of investigation jobs to their original groups -> #164988