QA (public) » openQA Project (public) » openQA Tests (public) » Containers and images

Alright, so I took a look at the current situation.

1. We have a Skopeo test module that is very runtime-agnostic. It should be able to run just fine on docker or on podman. https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/containers/skopeo.pm
2. We have the CONTAINER_RUNTIMES variable, which can take multiple values - e.g. podman,docker
3. I only see the Skopeo test being loaded here: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/0faff221ab2bc5a8d56dea28c550c72ea8536a29/lib/main_containers.pm#L145

So now I'm thinking about the implementation - the obvious way seems to be to make sure that passing both runtimes in the variables spins up two test runs, one for each runtime. I'm not sure how much effort that would be so I'm digging in to figure it out. Alternatively, a change in the codebase would make it so that we ALWAYS test on both runtimes, but that seems a bit hardcode-y

I scoured O3 for test cases but I couldn't find anything. Thankfully Martin Loviska sent me the exact test run I needed (and was kind enough to discuss my potential solution) : https://openqa.opensuse.org/tests/4397144# . So passing both values to the CONTAINER_RUNTIMES does indeed results in two executions.

These two executions however seem moot. Skopeo does not rely on any container runtime or daemon. Both inspect and copy seem to work fine without anything else installed.

In that case, the issue becomes a matter of "Do we test against newer Registry versions?". I went back to the Bugzilla and read it more thoroughly and from what I understand, the original issue was the packages in the OS itself were stale.

TL;DR; If we make sure we are running the Skopeo test against an Image Registry that is known to have the latest Registry API, the test is good enough.

After some further testing, I think the potential improvements to the test module would be:

1. Test against more than one registry (e.g. inspect something from Quay.io or Dockerhub.io.). Have to figure out how to get around the pull limit from public registries if we use those.
2. Spin up a local docker registry and test against it.
3. Get info on the registry we're using's version. There is not straightforward way to do that afaik.

We had a clarification meeting with the following outcome:

1. The issue in https://bugzilla.suse.com/show_bug.cgi?id=1219563 was that there a too old version of skopeo was incompatible with the API of a remote registry (unknown origin, could be docker.io, registry.suse.com, registry.opensuse.org, quay.io, ghcr.io, ...)
2. We can partly test for such issues by spinning up a own suse-registry container and test skopeo against that one
3. For point number 2 it is ok to modify skopeo.pm in such a way, that it required podman to be installed. This includes unscheduling skopeo.pm for any non-podman test runs.

After reading https://bugzilla.suse.com/show_bug.cgi?id=1219563#c21, it appears to me that the issue was the interaction between docker and buildah, and skopeo is unrelated:

15sp5docker:~ # docker run -it  --privileged -v /var/run/docker.sock:/var/run/docker.sock registry.suse.com/bci/bci-base:15.5
1d0a71969a4b:/ # zypper in -y skopeo buildah
1d0a71969a4b:/ # rpm -q skopeo buildah
skopeo-1.14.4-150300.11.11.1.x86_64
buildah-1.34.1-150500.3.7.1.x86_64
1d0a71969a4b:/ # sed -i 's/^driver =.*/driver="vfs"/' /etc/containers/storage.conf
1d0a71969a4b:/ # buildah from  docker-daemon:registry.suse.com/bci/bci-base:15.5
Error: initializing source docker-daemon:registry.suse.com/bci/bci-base:15.5: loading image from docker engine: Error response from daemon: client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

I'm not sure if this is something we should actually test, will ask for clarification: https://suse.slack.com/archives/C02DR0C5XUY/p1723723540616939.

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/20009