action #105145
Updated by livdywan over 2 years ago
## Observation
*osd-deployment* failed with [the same error](https://gitlab.suse.de/qa-maintenance/openQABot/-/pipelines/301728):
```
Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending
Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
[...]
Waiting for pod gitlab/runner-ydlpfvpg-project-3530-concurrent-0h2jfn to be running, status is Pending
908 ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
909 ContainersNotReady: "containers with unready status: [build helper]"
911ERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
```
## Acceptance criteria
* **AC1:** osd-deployment pipeline doesn't alert about internal gitlab k8s busy loops
* **AC2:** osd-deployment pipelines automatically restart on known internal errors
## Suggestions
* Look at the [upstream issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27367)
* File an infra ticket
* Come up with a way to silence alerts for internal GitLab errors