Pipeline of openQABot project fails with "urllib.error.HTTPError: HTTP Error 503: Service Unavailable" causing alert/notification
- See ticket description
- Restarting helped to let the job pass
- Pipeline: https://gitlab.suse.de/qa-maintenance/openQABot/-/pipelines/302125
- Job: https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/799531
It does not look like a retry was attempted. It is also not clear how the alert should be handled. Restarting the job helped to let it pass, not sure whether it is generally here required, though.
- AC1: The alert is actionable (e.g. there's some information about it on https://progress.opensuse.org/projects/qa/wiki#Alert-handling) or suppressed if there's nothing to be done.
- AC2: The actual problem is avoided if possible (e.g. by adding a retry)
- Status changed from In Progress to New
- Assignee deleted (
- Priority changed from Normal to Low
jbaier_cz you implemented retry for internal request calls in bbf4e04 . Looking at the callstack in https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/799531 it looks like now we have a problem within the osc python library. WDYT, should we do a custom retry loop or patch the osc library for retrying?
Yeah, looking at the stack
self.commentapi.get_comments(**kwargs) pretty much shows, that the error came from python-osc, probably we hit a moment where OBS was unresponsive or redeployed or something. It might be probably nicer, if we add the retry into the osc lib itself as we use it on many places (and other projects as well).
In the meantime as a workaround, we can probably automatically retrigger the pipeline itself. But I am not sure if this problems occurs so often to even bother ourselves (with the workaround).
- Status changed from In Progress to Feedback
Turned out that I do not want to touch the code much, the retry would be needed in osc/core.py which is using urllib.requests.urlopen() and some other stuff around. Instead I decided to create a wrapper for the http_GET function which is exposed to the comments.py. As I wanted to do a minimum to none changes to the comments.py, I kept the current XML parsing (which expects file handle to open).
A created https://gitlab.suse.de/qa-maintenance/openQABot/-/merge_requests/91 to address the issues.