action #105145
Updated by livdywan almost 3 years ago
## Observation *osd-deployment* failed with [the same error](https://gitlab.suse.de/qa-maintenance/openQABot/-/pipelines/301728): ``` Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending ContainersNotInitialized: "containers with incomplete status: [init-permissions]" ContainersNotReady: "containers with unready status: [build helper]" Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending ContainersNotInitialized: "containers with incomplete status: [init-permissions]" ContainersNotReady: "containers with unready status: [build helper]" [...] Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending 908 ContainersNotInitialized: "containers with incomplete status: [init-permissions]" 909 ContainersNotReady: "containers with unready status: [build helper]" 911ERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information ``` ## Acceptance criteria * **AC1:** osd-deployment pipeline doesn't alert about internal gitlab k8s busy loops * **AC2:** osd-deployment pipelines automatically restart on known internal errors ## Suggestions * Implement an automatic retry on error in the GitLab pipeline https://docs.gitlab.com/ee/ci/yaml/#retrywhen * Look at the [upstream issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27367) * File an infra ticket * Come up with a way to silence alerts for internal GitLab errors * Install our own GitLab runner on k8s