action #130952
closed[alert][thursday] gitlab.suse.de CI jobs fail with "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500" as of 2023-06-15 size:M
0%
Description
Observation¶
https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1634181 failed with
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500
fatal: expected flush after ref listing
I assume it's a current problem in Eng-Infra maintained infrastructure or OBS
Acceptance criteria¶
- AC1: Stable gitlab CI pipelines again with no errors about accessing other ressources like registry, gitlab, OBS/IBS, etc.
Suggestions¶
- Check for related gitlab CI pipeline failures
- If issue persists over "usual thursday maintenance window" problems then create SUSE-IT ticket
- For SUSE-IT ticket resolution ensure all our "usual" gitlab CI pipelines are stable again
Updated by okurz over 1 year ago
- Subject changed from [alert][thursday] gitlab.suse.de CI jobs fail with "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500" as of 2023-06-15 to [alert][thursday] gitlab.suse.de CI jobs fail with "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500" as of 2023-06-15 size:M
- Description updated (diff)
- Status changed from New to In Progress
https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs seems to be fine but there were other recent related failures, e.g. https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1636922
Updated by okurz over 1 year ago
- Status changed from In Progress to Workable
- Assignee deleted (
okurz)
Updated by okurz over 1 year ago
- Status changed from Workable to Resolved
- Assignee set to okurz
so we checked again on Monday. We have not received any gitlab CI failure messages over the weekend and also pages like https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs look clean. No further need for SUSE-IT SD ticket or something.
Updated by okurz over 1 year ago
- Status changed from Resolved to In Progress
happened again today outside of usual Thursday maintenance window in https://gitlab.suse.de/openqa/openqa-review/-/jobs/1656553 with
$ git clone --depth 1 https://github.com/os-autoinst/openqa_review.git
Cloning into 'openqa_review'...
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
fatal: expected flush after ref listing
Updated by okurz over 1 year ago
I added retry to https://build.opensuse.org/package/view_file/home:okurz:container/openqa-review/Dockerfile?expand=1 . I don't think an SUSE IT Eng-Infra ticket would help. We need to improve the retrying on our side.
Updated by okurz over 1 year ago
- Due date set to 2023-07-11
Updated by okurz over 1 year ago
I need to check again where the retry command should come from.
Updated by okurz over 1 year ago
- Status changed from Feedback to In Progress
oops, wrong container https://build.opensuse.org/package/view_file/home:okurz:container:ca/openqa-review/Dockerfile?expand=1 vs. https://build.opensuse.org/package/view_file/home:okurz:container/openqa-review/Dockerfile?expand=1 . Updated that now, waiting for https://build.opensuse.org/package/show/home:okurz:container:ca/openqa-review to publish again.
EDIT: Was published, retriggered https://gitlab.suse.de/openqa/openqa-review/-/jobs/1659541
Now failing because "getopt" is missing. Corrected in https://github.com/okurz/retry/commit/0230dc519b30ad6ddfaf81668dd1f0bac48a6b91, updated package and created submission with https://build.opensuse.org/request/show/1095796. Reverting the change in openqa-review pipelines for now with https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/18 (merged). Now waiting for SR to be accepted into Factory, new snapshot published, new container image built and then trying again.
Monitoring https://gitlab.suse.de/openqa/openqa-review/-/pipelines/716109, green again
Updated by okurz over 1 year ago
- Status changed from In Progress to Feedback
Needing to wait for https://build.opensuse.org/request/show/1095796 and according Tumbleweed snapshot published
Updated by okurz over 1 year ago
podman run --pull=newer --rm -it registry.opensuse.org/home/okurz/container/ca/containers/tumbleweed:openqa-review retry --help is fine, created https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/19, merged
Triggered https://gitlab.suse.de/openqa/openqa-review/-/pipelines/720301
Updated by okurz over 1 year ago
- Due date deleted (
2023-07-11) - Status changed from Feedback to Resolved
Updated by okurz over 1 year ago
- Status changed from Resolved to In Progress
ok, we have other gitlab CI jobs failing now. Like https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1671073
Updated by okurz over 1 year ago
- Status changed from In Progress to Blocked
Updated by okurz over 1 year ago
- Priority changed from Urgent to High
SD ticket was now being picked up and is worked on
Updated by okurz over 1 year ago
- Status changed from Blocked to Resolved
Should be fine. Meanwhile gitlab is running from PRG2 so fresh start with a new chance for stability :)