action #138377
open
bot-ng and openQABot pipelines fail to pull containers from registry.suse.de size:M
Added by livdywan 9 months ago.
Updated 9 days ago.
Description
Observation¶
https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1924326
ERROR: Job failed: failed to pull image "registry.suse.de/qa/maintenance/containers/qam-ci-leap:latest" with specified policies [always]: Error response from daemon: unknown: redis keepalive process: must be POST:/notify/redis request (manager.go:237:0s)
Same with registry.suse.de/qa/maintenance/containers/qam-ci-leap:latest in https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1924350
Acceptance Criteria¶
- AC1: Pipelines relying on registry.suse.de are known to work
Rollback steps¶
Suggestions¶
- Escalate this with OBS folks
- done, waiting on feedback
- Change pipeline scripts to run podman manually with retries so we can be more resilient to temporary issues
- Reach out to GitLab folks
- Tags set to infra
- Priority changed from Normal to Urgent
- Tags deleted (
infra)
- Description updated (diff)
- Priority changed from Urgent to Normal
- Status changed from New to In Progress
- Priority changed from Normal to Urgent
- Description updated (diff)
- Status changed from In Progress to Feedback
- Priority changed from Urgent to High
Pipelines inactive for now.
- Status changed from Feedback to In Progress
- Priority changed from High to Urgent
Pipelines have been run since but seem to fail on and off. Waiting to confirm if they are being fixed.
Maybe we need to reconsider the container setup. Although right now this affects several teams beyond maintenance.
Apparently the to my mind correct ticket state was misleading so I'm changing it. Either way I have been monitoring this.
If anyone has an idea and would like to jump in please do. Right now I physically can't test a change to CI. Alternatively feel free to retriever pipelines and see if they work by chance
- Due date set to 2023-11-08
Setting due date based on mean cycle time of SUSE QE Tools
- Status changed from In Progress to Feedback
- Priority changed from Urgent to High
livdywan wrote in #note-10:
Pipelines are consistently looking good. Unfortunately I still have no clue why https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipelines
This was brought up in email to autobuild@suse.de as suggested by OBS folks and includes other affected parties. Waiting on a response there.
Pipelines continue to look food. Still no clue what has or hasn't been fixed. Lowering priority accordingly.
- Description updated (diff)
- Subject changed from bot-ng pipelines fail to pull containers from registry.suse.de to bot-ng and qembot pipelines fail to pull containers from registry.suse.de
Well, we got another failure. I'm not opening another ticket since this is the same symptom but on openQABot and a new ticket for this would be a waste of time at this point:
ERROR: Job failed: failed to pull image "registry.opensuse.org/home/darix/apps/containers/gitlab-runner-helper:x86_64-latest" with specified policies [always]: Error response from daemon: received unexpected HTTP status: 503 Service Unavailable (manager.go:237:0s)
- Subject changed from bot-ng and qembot pipelines fail to pull containers from registry.suse.de to bot-ng and qembot pipelines fail to pull containers from registry.suse.de size:M
- Description updated (diff)
- Subject changed from bot-ng and qembot pipelines fail to pull containers from registry.suse.de size:M to bot-ng and openQABot pipelines fail to pull containers from registry.suse.de size:M
livdywan wrote in #note-17:
I was under the impression retries wouldn't cover a broken container setup but a comment made me re-read the docs. Maybe we should enable retries for bot-ng and openQABot afterall.
True, https://docs.gitlab.com/ee/ci/yaml/#retry says
script_failure: Retry when:
The script failed.
The runner failed to pull the Docker image. For docker, docker+machine, kubernetes executors.
that sounds promising
- Status changed from Feedback to Resolved
Let's assume this will work as expected. We won't be able to "test" this unless there's another outage so I'm resolving - if needed we can re-open.
- Copied to action #160802: bot-ng and openQABot pipelines fail to pull containers from registry.suse.de - again added
- Status changed from Resolved to Feedback
- Priority changed from High to Normal
https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipelines/1199291/builds shows that we indeed retry but the retries happen so fast that they are depleted before anything really changes (especially if the gitlab contains already fail to get up). Can we wait between them or would that conflict with future scheduled runs?
(also feel free to resolve again and copy this into a new ticket)
- Due date deleted (
2023-11-08)
- Status changed from Feedback to Blocked
Also available in: Atom
PDF