QA » openQA Project

Observation¶

osd-deployment failed with the same error:

Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending
Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending
    ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
    ContainersNotReady: "containers with unready status: [build helper]"
[...]
Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending
908 ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
909 ContainersNotReady: "containers with unready status: [build helper]"
911ERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Acceptance criteria¶

• AC1: osd-deployment pipeline doesn't alert about internal gitlab k8s busy loops
• AC2: osd-deployment pipelines automatically restart on known internal errors

Suggestions¶

• Implement an automatic retry on error in the GitLab pipeline https://docs.gitlab.com/ee/ci/yaml/#retrywhen
• Look at the upstream issue
• File an infra ticket
• Come up with a way to silence alerts for internal GitLab errors
• Install our own GitLab runner on k8s