https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842019-09-04T07:01:25ZopenSUSE Project Management ToolopenQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2410972019-09-04T07:01:25ZJERiveraMoyajeriveramoya@suse.com
<ul><li><strong>Copied from</strong> <i><a class="issue tracker-4 status-3 priority-6 priority-high2 closed" href="/issues/54902">action #54902</a>: openQA on osd fails at "incomplete" status when uploading, "502 response: Proxy Error"</i> added</li></ul> openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2411032019-09-04T07:01:44ZJERiveraMoyajeriveramoya@suse.com
<ul><li><strong>Copied from</strong> deleted (<i><a class="issue tracker-4 status-3 priority-6 priority-high2 closed" href="/issues/54902">action #54902</a>: openQA on osd fails at "incomplete" status when uploading, "502 response: Proxy Error"</i>)</li></ul> openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2411092019-09-04T07:03:44Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/55328">action #55328</a>: job is considered incomplete by openQA but worker still pushes updates so that "job is not considered dead"</i> added</li></ul> openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2411122019-09-04T07:04:52Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li><li><strong>Assignee</strong> set to <i>kraih</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Target version</strong> set to <i>Current Sprint</i></li></ul><p>as this is very likely caused by <a href="https://github.com/os-autoinst/openQA/pull/2270" class="external">https://github.com/os-autoinst/openQA/pull/2270</a> assigning to you to crosscheck.</p>
openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2411302019-09-04T07:33:37Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>New</i></li><li><strong>Assignee</strong> deleted (<del><i>kraih</i></del>)</li><li><strong>Target version</strong> deleted (<del><i>Current Sprint</i></del>)</li></ul><p>sorry, I am all wrong. the PR is <em>not</em> yet deployed.</p>
openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2411332019-09-04T08:03:31Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>openQA on osd fails with empty logs</i> to <i>openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logs</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li><li><strong>Priority</strong> changed from <i>High</i> to <i>Urgent</i></li><li><strong>Target version</strong> set to <i>Current Sprint</i></li></ul> openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2411872019-09-04T08:44:36Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-3 priority-lowest closed" href="/issues/41882">action #41882</a>: all arm worker die after some time</i> added</li></ul> openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2411962019-09-04T08:45:07Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/54128">action #54128</a>: [tools] openqaworker-arm-3 is broken</i> added</li></ul> openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2414062019-09-04T16:34:03Zokurzokurz@suse.com
<ul></ul><p>I stopped salt-minion and openqa-worker.target. It looks like /var/lib/openqa/pool is on the same partition as / . I don't know what changed or how it looked like in before. Probably pool should be on NVME as well. <code>systemctl cat openqa_nvme_prepare.service</code> creates the pool but does not seem to do anything with it. This looks similar to <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: tests incomplete with "No space left on device" openqaworker-arm-2 ran out of space in uploading ... (Resolved)" href="https://progress.opensuse.org/issues/53261">#53261</a> only about "pool", not "cache". Could it be we deleted the "pool" symlink by mistake and should use a bind mount as well? Probably to be done properly with salt.</p>
<ol>
<li>change to bind mount for all,</li>
<li>add that to salt</li>
<li>add -3 the same and monitor all three</li>
</ol>
<p>Done first two with <a href="https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/160" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/160</a></p>
openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2414812019-09-05T08:49:17Zokurzokurz@suse.com
<ul></ul><p>today we have some incompletes on aarch64 but seems like only openqaworker-arm-2. I disabled the worker target on the host and will retrigger incompletes. They should be picked up on openqaworker-arm-1. See e.g. <a href="https://openqa.suse.de/tests/3326179" class="external">https://openqa.suse.de/tests/3326179</a> from <a href="https://openqa.suse.de/tests/?&resultfilter=Incomplete" class="external">https://openqa.suse.de/tests/?&resultfilter=Incomplete</a></p>
openQA Project - action #56447: openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logshttps://progress.opensuse.org/issues/56447?journal_id=2418742019-09-07T21:35:51Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>all problems resolved. The nvme preparation is done as available in salt and a workaround for nscd is applied, see <a href="https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/162" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/162</a> . The worker was able to successfully test build 0307 of SLES12SP5 so we should be good.</p>