https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-01-28T11:36:48ZopenSUSE Project Management ToolopenQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3807652021-01-28T11:36:48Zmkittlermarius.kittler@suse.com
<ul><li><strong>Assignee</strong> set to <i>mkittler</i></li></ul><p>This feature is actually orthogonal to the related epic. The only common part is the use of df/statfs. As discussed for the other ticket, we should make that part configurable at some point.</p>
<hr>
<p>I suppose I would create a separate script for this which checks df/statfs (sharing existing code introduced by <a href="https://github.com/os-autoinst/openQA/pull/3635" class="external">https://github.com/os-autoinst/openQA/pull/3635</a>). This script would enqueue the cleanup tasks depending on the outcome of the check. The script itself would be invoked by a systemd timer (which a higher frequency than the current timer which enqueues the cleanup unconditionally).</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3808472021-01-28T15:05:47Zokurzokurz@suse.com
<ul></ul><p>mkittler wrote:</p>
<blockquote>
<p>This feature is actually orthogonal to the related epic.</p>
</blockquote>
<p>Please explain further what you mean by that. And please do not just point to "what I wrote" :) Is this user story not leading into the same use case?</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3808572021-01-28T15:26:10Zmkittlermarius.kittler@suse.com
<ul></ul><p>Your wording "split out" makes it sound like you've took a big ticket and split the task up. However, this ticket seems to be a different task. The only thing in common with the task from the epic is that df/statfs() is involved. I have not been talking about any use-cases here so far but simply about what needs to be done to fulfill the ACs of the tickets.</p>
<p>However, I also don't see the use case for this ticket. Just triggering the current cleanup logic when running out of disk space will not necessarily prevent us from running out of disk space so it isn't really helpful in that regard. This feature could lower the frequency the cleanup is triggered. However, there's no practical benefit. If the cleanup it triggered more often than necessary that shouldn't cause any harm.</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3808592021-01-28T15:28:48Zmkittlermarius.kittler@suse.com
<ul></ul><p>Another hint: Coming up with further ideas how to use df/statfs() does not help implementing your previous ideas/tickets. It means <em>additional</em> work, not splitting up the work which is already planned or in progress.</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3808892021-01-28T16:49:24Zokurzokurz@suse.com
<ul></ul><p>Actually I think that this story could help to cover <a class="issue tracker-6 status-1 priority-3 priority-lowest child parent" title="coordination: [epic] Reconsider triggering cleanup jobs (New)" href="https://progress.opensuse.org/issues/64881">#64881</a> . When we trigger cleanup whenever there is need to cleanup then we do not need to trigger the complete cleanup by timer but instead just call the cheaper "df", right? And the task to call "df" could be done by a periodic minion job instead of a systemd timer so that this functionality needs no further system administration by the user and can work in systemd-less variants as well. The second benefit would be by checking more often that the cleanup hysteresis curve would be smaller and hence less problems with "flickering alerts" in grafana, right?</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3809372021-01-29T04:13:26Zopenqa_reviewopenqa-review@suse.de
<ul><li><strong>Due date</strong> set to <i>2021-02-12</i></li></ul><p>Setting due date based on mean cycle time of SUSE QE Tools</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3817732021-02-05T15:06:41Zmkittlermarius.kittler@suse.com
<ul></ul><blockquote>
<p>by a periodic minion job instead of a systemd timer</p>
</blockquote>
<p>We usually use systemd timers to <em>enqueue</em> Minion jobs so they run periodically. I don't know whether it is possible within the Minion framework itself to enqueue jobs periodically. (We can not do that from the main web UI service because it uses preforking. Technically we could enqueue Minion jobs periodically from any of the other services which do not use preforking.)</p>
<blockquote>
<p>When we trigger cleanup whenever there is need to cleanup then we do not need to trigger the complete cleanup by timer but instead just call the cheaper "df", right?</p>
</blockquote>
<p>This ticket would certainly help with <a class="issue tracker-6 status-1 priority-3 priority-lowest child parent" title="coordination: [epic] Reconsider triggering cleanup jobs (New)" href="https://progress.opensuse.org/issues/64881">#64881</a>. However, in the description it would help progressing with <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] Automatically remove assets+results based on available free space (New)" href="https://progress.opensuse.org/issues/76984">#76984</a> which is a whole different story.</p>
<blockquote>
<p>The second benefit would be by checking more often that the cleanup hysteresis curve would be smaller and hence less problems with "flickering alerts" in grafana, right?</p>
</blockquote>
<p>Yes, but again that's not really helping with <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] Automatically remove assets+results based on available free space (New)" href="https://progress.opensuse.org/issues/76984">#76984</a> which is for extending the cleanup algorithm (to solve running out of disk space due to over-allocation) and this ticket is for just triggering it more often (also to solve running out of disk space but due to running the cleanup not frequently enough and in addition to save resources by avoiding unnecessary cleanups). So these tickets address different problems.</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3817742021-02-05T15:09:20Zmkittlermarius.kittler@suse.com
<ul><li><strong>Assignee</strong> deleted (<del><i>mkittler</i></del>)</li></ul><p>I can assign the ticket to me again when we cleared up the questions.</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3828102021-02-15T17:27:33Zlivdywanliv.dywan@suse.com
<ul><li><strong>Due date</strong> deleted (<del><i>2021-02-12</i></del>)</li></ul><p>FYI: Unsetting the Due date since that only makes sense when someone's actively working on it.</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3845592021-02-22T10:46:18Zokurzokurz@suse.com
<ul></ul><p>Discussed with mkittler: Next steps</p>
<ul>
<li>Add "early-abort" in cleanup jobs based on df-output with configurable minimum space to keep free. With this we re-use the existing systemd time based triggering for the space-aware check</li>
<li>When we have the early-abort we can configure the systemd timer to trigger more often (instance based) to reduce the hysteresis size</li>
<li>Trigger cleanup also when creating new jobs, see <a class="issue tracker-6 status-1 priority-3 priority-lowest child parent" title="coordination: [epic] Reconsider triggering cleanup jobs (New)" href="https://progress.opensuse.org/issues/64881">#64881</a></li>
</ul>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3847332021-02-22T15:11:00Zmkittlermarius.kittler@suse.com
<ul><li><strong>Assignee</strong> set to <i>mkittler</i></li></ul> openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3848562021-02-23T04:06:16Zopenqa_reviewopenqa-review@suse.de
<ul><li><strong>Due date</strong> set to <i>2021-03-09</i></li></ul><p>Setting due date based on mean cycle time of SUSE QE Tools</p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3850752021-02-23T14:21:19Zmkittlermarius.kittler@suse.com
<ul></ul><p>Draft PR: <a href="https://github.com/os-autoinst/openQA/pull/3750" class="external">https://github.com/os-autoinst/openQA/pull/3750</a></p>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3850782021-02-23T14:21:30Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li></ul> openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3857082021-02-25T13:58:40Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><ul>
<li>PR has been merged</li>
<li>SR for OSD: <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/456" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/456</a></li>
<li>Configured the same on o3 manually</li>
</ul>
openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3873462021-03-03T17:07:46Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>It seems to work in production:</p>
<p>OSD:</p>
<pre><code> "result" => "Skipping, free disk space on '/var/lib/openqa/share/factory' exceeds configured percentage 20 % (free percentage: 22.9447968855517 %)",
</code></pre>
<p>(result cleanup not skipped)</p>
<pre><code>martchus@openqa:~> df -h /assets /results
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
/dev/vdc 7,0T 5,5T 1,6T 79% /assets
/dev/vdd 5,5T 4,5T 1,1T 81% /results
</code></pre>
<p>o3:</p>
<pre><code> "result" => "Skipping, free disk space on '/var/lib/openqa/share/factory' exceeds configured percentage 20 % (free percentage: 29.5415074793063 %)",
"result" => "Skipping, free disk space on '/var/lib/openqa/testresults' exceeds configured percentage 20 % (free percentage: 45.2762918429145 %)",
</code></pre><pre><code>martchus@ariel:~> df -h /assets /var/lib/openqa/testresults
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
/dev/mapper/vg0-assets 3,0T 2,2T 894G 71% /assets
/dev/vdb1 5,0T 2,8T 2,3T 56% /var/lib/openqa
</code></pre> openQA Project - action #88121: Trigger cleanup of results (or assets) if not enough free space based on configuration limithttps://progress.opensuse.org/issues/88121?journal_id=3927292021-03-18T13:15:18Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> deleted (<del><i>2021-03-09</i></del>)</li></ul>