Project

General

Profile

Actions
Copy link

action #179260

closed

[Containers] test fails in run_container_in_k8s_GCE. Container Registry is deprecated and shutting down

Added by asmorodskyi 2 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
mloviska
Target version:
-
Start date:
2025-03-20
Due date:
% Done:

0%

Estimated time:
Tags:
containers, bci, k8s

Description

Observation

openQA test in scenario sle-15-SP6-BCI-Updates-x86_64-php-fpm_8_on_k8s@64bit fails in
run_container_in_k8s_GCE

ERROR: (gcloud.container.images.delete) Bad status during token exchange: 412
b"Container Registry is deprecated and shutting down, please use the auto migration tool to migrate to Artifact Registry (gcloud artifacts docker upgrade migrate --projects='ei-sle-qa-4a51'). For more details see: https://cloud.google.com/artifact-registry/docs/transition/auto-migrate-gcr-ar"

Reproducible

Fails since (at least) Build 48.43_php-fpm8-image

Expected result

Last good: 48.41_php-fpm8-image (or more recent)

Further details

Always latest result in this scenario: latest

Actions
Copy link
 #1

Updated by asmorodskyi 2 months ago

  • Project changed from openQA Tests (public) to Containers and images
  • Category deleted (Bugs in existing tests)
  • Priority changed from Normal to High
Actions
Copy link
 #2

Updated by mloviska 2 months ago

  • Status changed from New to In Progress
  • Assignee set to mloviska
Actions
Copy link
 #3

Updated by mloviska 2 months ago

The delete command seemingly worked well

gcloud container images list  --repository=gcr.io/ei-sle-qa-4a51
Listed 0 items.
Actions
Copy link
 #4

Updated by mloviska 2 months ago

Alright, taking back the previous statement, there are over 600 images in the registry that we should clean up.

❯ gcloud container images delete eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f --force-delete-tags
Digests:
- eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f
  Associated tags:
 - latest
This operation will delete the tags and images identified by the digests above.

Do you want to continue (Y/n)?  Y

Deleted [eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927:latest].
Deleted [eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f].
❯ gcloud container images describe eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f
ERROR: (gcloud.container.images.describe) [eu.gcr.io/ei-sle-qa-4a51/openqa-1318892713188927@sha256:c1231d160048eb1a7464ef556dad7b5a48df79100b507807333e0000b7cd956f] is not found or is not a valid name. Expected tag in the form "base:tag" or "tag" or digest in the form "sha256:<digest>"

This apparently worked

Actions
Copy link
 #5

Updated by mloviska 2 months ago

The current cleanup works, so we have a bunch of old images lying there

Actions
Copy link
 #6

Updated by mloviska 2 months ago

ofc, the command from c#3 addressed wrong repo

gcloud container images list --repository=eu.gcr.io/ei-sle-qa-4a51 | wc -l
698
Actions
Copy link
 #7

Updated by mloviska 2 months ago

Running a reproducer while cleaning the registry, I see the registry occasionally responds with Bad status during token exchange: 412

Actions
Copy link
 #8

Updated by mloviska 2 months ago 

gcloud artifacts docker upgrade migrate --projects=[redacted] --no-use-analyze-iam --skip-iam-update
gcloud container images list-gcr-usage --project=ei-sle-qa-4a51
---
repository: eu.gcr.io/ei-sle-qa-4a51
usage: REDIRECTED

Let's see whether is going to work properly

Actions
Copy link
 #9

Updated by mloviska 2 months ago

  • Status changed from In Progress to Feedback

So far the tests are passing and I can see the images in the Artifact Registry. As of now, the old code works too.

Actions
Copy link
 #10

Updated by ph03nix 2 months ago

  • Tags set to containers, bci, k8s
  • Subject changed from test fails in run_container_in_k8s_GCE. Container Registry is deprecated and shutting down to [Containers] test fails in run_container_in_k8s_GCE. Container Registry is deprecated and shutting down
Actions
Copy link
 #11

Updated by mloviska 2 months ago

  • Status changed from Feedback to Resolved
Actions
Copy link

Also available in: Atom PDF