https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842022-01-20T15:25:14ZopenSUSE Project Management ToolopenQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4822122022-01-20T15:25:14Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li><li><strong>Target version</strong> set to <i>Ready</i></li></ul><p>I will look into retrying</p>
openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4822242022-01-20T15:26:35Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/95822">action #95822</a>: qa-maintenance/openQABot failed to trigger aggregate tests with "urllib.error.HTTPError: HTTP Error 500: Internal Server Error"</i> added</li></ul> openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4822302022-01-20T15:26:42Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-1 priority-4 priority-default" href="/issues/101274">action #101274</a>: openQABot pipeline failed with NewConnectionError</i> added</li></ul> openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4822362022-01-20T15:32:04Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed" href="/issues/104085">action #104085</a>: openQABot pipeline failed with terminating connection due to administrator command size:S</i> added</li></ul> openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4822512022-01-20T15:38:29Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>New</i></li><li><strong>Assignee</strong> deleted (<del><i>okurz</i></del>)</li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Low</i></li></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/29611">@jbaier_cz</a> you implemented retry for internal request calls in bbf4e04 . Looking at the callstack in <a href="https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/799531" class="external">https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/799531</a> it looks like now we have a problem within the osc python library. WDYT, should we do a custom retry loop or patch the osc library for retrying?</p>
openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4822962022-01-20T19:03:44Zjbaier_czjbaier@suse.cz
<ul></ul><p>Yeah, looking at the stack <code>self.commentapi.get_comments(**kwargs)</code> pretty much shows, that the error came from python-osc, probably we hit a moment where OBS was unresponsive or redeployed or something. It might be probably nicer, if we add the retry into the osc lib itself as we use it on many places (and other projects as well).</p>
<p>In the meantime as a workaround, we can probably automatically retrigger the pipeline itself. But I am not sure if this problems occurs so often to even bother ourselves (with the workaround).</p>
openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4865282022-02-03T09:28:20Zjbaier_czjbaier@suse.cz
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>jbaier_cz</i></li></ul> openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4869362022-02-03T16:30:18Zjbaier_czjbaier@suse.cz
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>Turned out that I do not want to touch the code much, the retry would be needed in osc/core.py which is using urllib.requests.urlopen() and some other stuff around. Instead I decided to create a wrapper for the http_GET function which is exposed to the comments.py. As I wanted to do a minimum to none changes to the comments.py, I kept the current XML parsing (which expects file handle to open).</p>
<p>A created <a href="https://gitlab.suse.de/qa-maintenance/openQABot/-/merge_requests/91" class="external">https://gitlab.suse.de/qa-maintenance/openQABot/-/merge_requests/91</a> to address the issues.</p>
openQA Infrastructure - action #105169: Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notificationhttps://progress.opensuse.org/issues/105169?journal_id=4897672022-02-10T20:24:59Zjbaier_czjbaier@suse.cz
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>Code is merged and running, so far without issues. In the future, we should see more detailed info about the problem and also it should auto-retry first.</p>