action #179260
closed
[Containers] test fails in run_container_in_k8s_GCE. Container Registry is deprecated and shutting down
Added by asmorodskyi 22 days ago.
Updated 13 days ago.
Description
Observation¶
openQA test in scenario sle-15-SP6-BCI-Updates-x86_64-php-fpm_8_on_k8s@64bit fails in
run_container_in_k8s_GCE
ERROR: (gcloud.container.images.delete) Bad status during token exchange: 412
b"Container Registry is deprecated and shutting down, please use the auto migration tool to migrate to Artifact Registry (gcloud artifacts docker upgrade migrate --projects='ei-sle-qa-4a51'). For more details see: https://cloud.google.com/artifact-registry/docs/transition/auto-migrate-gcr-ar"
Reproducible¶
Fails since (at least) Build 48.43_php-fpm8-image
Expected result¶
Last good: 48.41_php-fpm8-image (or more recent)
Further details¶
Always latest result in this scenario: latest
- Project changed from openQA Tests (public) to Containers and images
- Category deleted (
Bugs in existing tests)
- Priority changed from Normal to High
- Status changed from New to In Progress
- Assignee set to mloviska
The delete command seemingly worked well
gcloud container images list --repository=gcr.io/ei-sle-qa-4a51
Listed 0 items.
Alright, taking back the previous statement, there are over 600 images in the registry that we should clean up.
❯ gcloud container images delete eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f --force-delete-tags
Digests:
- eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f
Associated tags:
- latest
This operation will delete the tags and images identified by the digests above.
Do you want to continue (Y/n)? Y
Deleted [eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927:latest].
Deleted [eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f].
❯ gcloud container images describe eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f
ERROR: (gcloud.container.images.describe) [eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f] is not found or is not a valid name. Expected tag in the form "base:tag" or "tag" or digest in the form "sha256:<digest>"
This apparently worked
The current cleanup works, so we have a bunch of old images lying there
ofc, the command from c#3 addressed wrong repo
gcloud container images list --repository=eu.gcr.io/ei-sle-qa-4a51 | wc -l
698
Running a reproducer while cleaning the registry, I see the registry occasionally responds with Bad status during token exchange: 412
gcloud artifacts docker upgrade migrate --projects=[redacted] --no-use-analyze-iam --skip-iam-update
gcloud container images list-gcr-usage --project=ei-sle-qa-4a51
---
repository: eu.gcr.io/ei-sle-qa-4a51
usage: REDIRECTED
Let's see whether is going to work properly
- Status changed from In Progress to Feedback
So far the tests are passing and I can see the images in the Artifact Registry. As of now, the old code works too.
- Tags set to containers, bci, k8s
- Subject changed from test fails in run_container_in_k8s_GCE. Container Registry is deprecated and shutting down to [Containers] test fails in run_container_in_k8s_GCE. Container Registry is deprecated and shutting down
- Status changed from Feedback to Resolved
Also available in: Atom
PDF