https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842022-11-02T13:27:51ZopenSUSE Project Management ToolopenQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5700072022-11-02T13:27:51Zdheidlerdheidler@suse.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5700102022-11-02T13:34:02Zokurzokurz@suse.com
<ul><li><strong>Target version</strong> set to <i>Ready</i></li></ul> openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5703192022-11-03T10:18:51Zmkittlermarius.kittler@suse.com
<ul><li><strong>Subject</strong> changed from <i>Failed pipeline for "openqa-worker" in salt-states-opensuse</i> to <i>Failed pipeline for "openqa-worker" in salt-states-opensuse size:M</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/570319/diff?detail_id=536761">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li></ul> openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5709222022-11-04T12:54:03Zdheidlerdheidler@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>dheidler</i></li></ul><p>This looks like a repo issue or an issue regarding local copy of repo metadata being out of date.<br>
PR as suggested: <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/765" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/765</a></p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5709252022-11-04T12:54:18Zdheidlerdheidler@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul> openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5723892022-11-09T08:25:42Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> set to <i>2022-11-18</i></li></ul> openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5741942022-11-14T12:46:44Zlivdywanliv.dywan@suse.com
<ul></ul><p>dheidler wrote:</p>
<blockquote>
<p>This looks like a repo issue or an issue regarding local copy of repo metadata being out of date.<br>
PR as suggested: <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/765" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/765</a></p>
</blockquote>
<p>This is still under review. Might be worth discussing with others since I feel like Dominik was expecting a more trivial fix.</p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5758682022-11-17T19:12:38Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>Failed pipeline for "openqa-worker" in salt-states-opensuse size:M</i> to <i>Failed pipeline for "openqa-worker" in salt-states-opensuse</i></li><li><strong>Due date</strong> deleted (<del><i>2022-11-18</i></del>)</li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>New</i></li><li><strong>Assignee</strong> deleted (<del><i>dheidler</i></del>)</li></ul><p>cdywan wrote:</p>
<blockquote>
<p>dheidler wrote:</p>
<blockquote>
<p>This looks like a repo issue or an issue regarding local copy of repo metadata being out of date.<br>
PR as suggested: <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/765" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/765</a></p>
</blockquote>
<p>This is still under review. Might be worth discussing with others since I feel like Dominik was expecting a more trivial fix.</p>
</blockquote>
<p>Then we need to rediscuss although I think the original ticket description already covers it:</p>
<blockquote>
<p>We already have instructed salt to call zypper multiple times for retry. But it looks like the repository data is not refreshed between each call. So we need to ensure that also the refreshing is done multiple times. In <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls#L8" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls#L8</a> we say "refresh: False" to save time but here it does not help us. So we should check if we change back to refresh how long it takes in comparison.</p>
</blockquote>
<p>meaning: It's not as simple is just putting "refresh: True" there. Also it wouldn't be "size:M" if it's just that, right?</p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5762042022-11-18T11:17:53Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>Failed pipeline for "openqa-worker" in salt-states-opensuse</i> to <i>Failed pipeline for "openqa-worker" in salt-states-opensuse size:M</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/576204/diff?detail_id=542295">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li></ul> openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5770302022-11-22T15:04:42Zmkittlermarius.kittler@suse.com
<ul><li><strong>Assignee</strong> set to <i>mkittler</i></li></ul> openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5770332022-11-22T15:16:56Zmkittlermarius.kittler@suse.com
<ul></ul><p>According to the documentation <a href="https://docs.saltproject.io/en/latest/ref/states/all/salt.states.pkg.html" class="external">https://docs.saltproject.io/en/latest/ref/states/all/salt.states.pkg.html</a> using <code>refresh: True</code> will slow us down as we have multiple <code>pkg</code> states and then a refresh would be done for all of them. I can nevertheless create a MR to see how bad it'll be. Keeping Salt's default might not be helpful. At least the documentation doesn't state that then a refresh would be done in case a retry is done. Neither the mentioned documentation nor <a href="https://docs.saltproject.io/en/latest/ref/states/requisites.html#retrying-states" class="external">https://docs.saltproject.io/en/latest/ref/states/requisites.html#retrying-states</a> describe the interaction between <code>refresh</code> and <code>retry</code>. I'm also not sure how we would test ourselves how the behavior. We'd somehow needed to provoke the error and somehow trace whether a refresh is done.</p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5770362022-11-22T15:25:44Zmkittlermarius.kittler@suse.com
<ul></ul><p>MR: <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/776" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/776</a></p>
<p>CI runtimes on master (with <code>refresh: False</code>):</p>
<ul>
<li>test-storage: 00:02:46</li>
<li>test-monitor: 00:03:38</li>
<li>test-worker: 00:12:30</li>
<li>test-webui: 00:05:28</li>
</ul>
<p>CI runtimes with <code>refresh: True</code>:</p>
<ul>
<li>test-storage: 00:03:37</li>
<li>test-monitor: 00:05:14</li>
<li>test-worker: 00:12:30</li>
<li>test-webui: 00:08:10</li>
</ul>
<p>So it generally takes a few minutes longer. Strangely <code>test-worker</code> had the same runtime. Not sure whether that's acceptable.</p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5778462022-11-24T15:37:56Zmkittlermarius.kittler@suse.com
<ul><li><strong>Assignee</strong> deleted (<del><i>mkittler</i></del>)</li></ul><p>I currently have enough tickets assigned. Maybe I'll pick this one up later. It would also make sense to discuss the outcome of my test (mentioned in the previous comment).</p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5807772022-12-02T15:22:13Zdheidlerdheidler@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>Feedback</i></li><li><strong>Assignee</strong> set to <i>dheidler</i></li></ul><p>I personally would consider everything below 15 minutes as acceptable - especially as it saves us time reacting on issues.</p>
<p>So I would go for merging this.<br>
Any objections?</p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5807892022-12-02T16:25:11Zokurzokurz@suse.com
<ul></ul><p>Well, as mkittler tested the runtime does increase but not for the worker. However the additional time is not only necessary during CI runs but any time someone or a service tries to apply a salt high state which I consider significant. As we need to do some retrying anyway I would favor if we find a more efficient solution that tries the fastest way first and only refresh in any retries as necessary</p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5810052022-12-05T12:22:58Zdheidlerdheidler@suse.com
<ul></ul><p>Hm - we could set retry to true maybe with some env var that is only set when the pipeline is applied from gitlab. WDYT?<br>
I don't know any way how (or even if) your idea could be achieved using salt.</p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5821092022-12-07T09:08:18Zokurzokurz@suse.com
<ul></ul><p>dheidler wrote:</p>
<blockquote>
<p>Hm - we could set retry to true maybe with some env var that is only set when the pipeline is applied from gitlab. WDYT?<br>
I don't know any way how (or even if) your idea could be achieved using salt.</p>
</blockquote>
<p>This brought me to an idea: When we only want to effectively "retry" when running in CI jobs then let's do that, but not "refresh" but simply CI level retry:<br>
<a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/778" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/778</a></p>
openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5823072022-12-07T10:57:31Zmkittlermarius.kittler@suse.com
<ul><li><strong>Subject</strong> changed from <i>Failed pipeline for "openqa-worker" in salt-states-opensuse size:M</i> to <i>Failed pipeline for "openqa-worker" in salt-states-openqa size:M</i></li></ul> openQA Infrastructure - action #119767: Failed pipeline for "openqa-worker" in salt-states-openqa size:Mhttps://progress.opensuse.org/issues/119767?journal_id=5824392022-12-07T15:00:36Zdheidlerdheidler@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>Let's see if it happens again:<br>
<a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/pipelines/545047" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/pipelines/545047</a></p>