Project

General

Profile

Actions
Copy link

action #130952

closed

[alert][thursday] gitlab.suse.de CI jobs fail with "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500" as of 2023-06-15 size:M

Added by okurz 11 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-06-15
Due date:
% Done:

0%

Estimated time:
Tags:
alert, gitlab, infra, thursday

Description

Observation

https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1634181 failed with

error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500
fatal: expected flush after ref listing

I assume it's a current problem in Eng-Infra maintained infrastructure or OBS

Acceptance criteria

  • AC1: Stable gitlab CI pipelines again with no errors about accessing other ressources like registry, gitlab, OBS/IBS, etc.

Suggestions

  • Check for related gitlab CI pipeline failures
  • If issue persists over "usual thursday maintenance window" problems then create SUSE-IT ticket
  • For SUSE-IT ticket resolution ensure all our "usual" gitlab CI pipelines are stable again
Actions
Copy link
 #1

Updated by okurz 11 months ago

  • Subject changed from [alert][thursday] gitlab.suse.de CI jobs fail with "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500" as of 2023-06-15 to [alert][thursday] gitlab.suse.de CI jobs fail with "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500" as of 2023-06-15 size:M
  • Description updated (diff)
  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by okurz 11 months ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)
Actions
Copy link
 #3

Updated by okurz 11 months ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz

so we checked again on Monday. We have not received any gitlab CI failure messages over the weekend and also pages like https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs look clean. No further need for SUSE-IT SD ticket or something.

Actions
Copy link
 #4

Updated by okurz 11 months ago

  • Status changed from Resolved to In Progress

happened again today outside of usual Thursday maintenance window in https://gitlab.suse.de/openqa/openqa-review/-/jobs/1656553 with

$ git clone --depth 1 https://github.com/os-autoinst/openqa_review.git
Cloning into 'openqa_review'...
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
fatal: expected flush after ref listing
Actions
Copy link
 #5

Updated by okurz 11 months ago

I added retry to https://build.opensuse.org/package/view_file/home:okurz:container/openqa-review/Dockerfile?expand=1 . I don't think an SUSE IT Eng-Infra ticket would help. We need to improve the retrying on our side.

Actions
Copy link
 #7

Updated by okurz 11 months ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #8

Updated by okurz 11 months ago

I need to check again where the retry command should come from.

Actions
Copy link
 #9

Updated by okurz 11 months ago

  • Status changed from Feedback to In Progress

oops, wrong container https://build.opensuse.org/package/view_file/home:okurz:container:ca/openqa-review/Dockerfile?expand=1 vs. https://build.opensuse.org/package/view_file/home:okurz:container/openqa-review/Dockerfile?expand=1 . Updated that now, waiting for https://build.opensuse.org/package/show/home:okurz:container:ca/openqa-review to publish again.

EDIT: Was published, retriggered https://gitlab.suse.de/openqa/openqa-review/-/jobs/1659541

Now failing because "getopt" is missing. Corrected in https://github.com/okurz/retry/commit/0230dc519b30ad6ddfaf81668dd1f0bac48a6b91, updated package and created submission with https://build.opensuse.org/request/show/1095796. Reverting the change in openqa-review pipelines for now with https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/18 (merged). Now waiting for SR to be accepted into Factory, new snapshot published, new container image built and then trying again.

Monitoring https://gitlab.suse.de/openqa/openqa-review/-/pipelines/716109, green again

Actions
Copy link
 #10

Updated by okurz 11 months ago

  • Status changed from In Progress to Feedback

Needing to wait for https://build.opensuse.org/request/show/1095796 and according Tumbleweed snapshot published

Actions
Copy link
 #11

Updated by okurz 10 months ago

podman run --pull=newer --rm -it registry.opensuse.org/home/okurz/container/ca/containers/tumbleweed:openqa-review retry --help is fine, created https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/19, merged

Triggered https://gitlab.suse.de/openqa/openqa-review/-/pipelines/720301

Actions
Copy link
 #12

Updated by okurz 10 months ago

  • Due date deleted (2023-07-11)
  • Status changed from Feedback to Resolved
Actions
Copy link
 #13

Updated by okurz 10 months ago

  • Status changed from Resolved to In Progress

ok, we have other gitlab CI jobs failing now. Like https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1671073

Actions
Copy link
 #14

Updated by okurz 10 months ago

  • Status changed from In Progress to Blocked
Actions
Copy link
 #15

Updated by okurz 10 months ago

  • Priority changed from Urgent to High

SD ticket was now being picked up and is worked on

Actions
Copy link
 #16

Updated by okurz 9 months ago

  • Status changed from Blocked to Resolved

Should be fine. Meanwhile gitlab is running from PRG2 so fresh start with a new chance for stability :)

Actions
Copy link

Also available in: Atom PDF