action #163553
closedAdd API compatability test to skopeo
0%
Description
The current skopeo
test runs test only skopeo in it's default configuration. https://bugzilla.suse.com/show_bug.cgi?id=1219563 shows that we should also test skopeo in conjunction with docker to detect such issues in the test runs.
This ticket is about extending the current skopeo test runs in such a way, that it can run with the podman (should remain the default) and with the docker daemon.
Acceptance criteria¶
- Modify
skopeo.pm
such, that it uses the docker daemon for docker test runs - Add the skopeo tests to the docker test runs
Updated by ph03nix 7 months ago
- Related to action #154933: Create test for skopeo added
Updated by hilchev 6 months ago
Alright, so I took a look at the current situation.
- We have a Skopeo test module that is very runtime-agnostic. It should be able to run just fine on docker or on podman. https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/containers/skopeo.pm
- We have the
CONTAINER_RUNTIMES
variable, which can take multiple values - e.g.podman,docker
- I only see the Skopeo test being loaded here: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/0faff221ab2bc5a8d56dea28c550c72ea8536a29/lib/main_containers.pm#L145
So now I'm thinking about the implementation - the obvious way seems to be to make sure that passing both runtimes in the variables spins up two test runs, one for each runtime. I'm not sure how much effort that would be so I'm digging in to figure it out. Alternatively, a change in the codebase would make it so that we ALWAYS test on both runtimes, but that seems a bit hardcode-y
Updated by hilchev 6 months ago
I scoured O3 for test cases but I couldn't find anything. Thankfully Martin Loviska sent me the exact test run I needed (and was kind enough to discuss my potential solution) : https://openqa.opensuse.org/tests/4397144# . So passing both values to the CONTAINER_RUNTIMES
does indeed results in two executions.
These two executions however seem moot. Skopeo does not rely on any container runtime or daemon. Both inspect
and copy
seem to work fine without anything else installed.
In that case, the issue becomes a matter of "Do we test against newer Registry versions?". I went back to the Bugzilla and read it more thoroughly and from what I understand, the original issue was the packages in the OS itself were stale.
TL;DR; If we make sure we are running the Skopeo test against an Image Registry that is known to have the latest Registry API, the test is good enough.
Updated by hilchev 6 months ago
After some further testing, I think the potential improvements to the test module would be:
- Test against more than one registry (e.g. inspect something from Quay.io or Dockerhub.io.). Have to figure out how to get around the pull limit from public registries if we use those.
- Spin up a local docker registry and test against it.
- Get info on the registry we're using's version. There is not straightforward way to do that afaik.
Updated by ph03nix 6 months ago
We had a clarification meeting with the following outcome:
- The issue in https://bugzilla.suse.com/show_bug.cgi?id=1219563 was that there a too old version of skopeo was incompatible with the API of a remote registry (unknown origin, could be docker.io, registry.suse.com, registry.opensuse.org, quay.io, ghcr.io, ...)
- We can partly test for such issues by spinning up a own suse-registry container and test skopeo against that one
- For point number 2 it is ok to modify
skopeo.pm
in such a way, that it requiredpodman
to be installed. This includes unschedulingskopeo.pm
for any non-podman test runs.
Updated by ph03nix 6 months ago ยท Edited
After reading https://bugzilla.suse.com/show_bug.cgi?id=1219563#c21, it appears to me that the issue was the interaction between docker
and buildah
, and skopeo is unrelated:
15sp5docker:~ # docker run -it --privileged -v /var/run/docker.sock:/var/run/docker.sock registry.suse.com/bci/bci-base:15.5
1d0a71969a4b:/ # zypper in -y skopeo buildah
1d0a71969a4b:/ # rpm -q skopeo buildah
skopeo-1.14.4-150300.11.11.1.x86_64
buildah-1.34.1-150500.3.7.1.x86_64
1d0a71969a4b:/ # sed -i 's/^driver =.*/driver="vfs"/' /etc/containers/storage.conf
1d0a71969a4b:/ # buildah from docker-daemon:registry.suse.com/bci/bci-base:15.5
Error: initializing source docker-daemon:registry.suse.com/bci/bci-base:15.5: loading image from docker engine: Error response from daemon: client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version
I'm not sure if this is something we should actually test, will ask for clarification: https://suse.slack.com/archives/C02DR0C5XUY/p1723723540616939.
Updated by ph03nix 5 months ago
- Status changed from Workable to Closed
See https://suse.slack.com/archives/C02DR0C5XUY/p1725021286474529?thread_ts=1723723540.616939&cid=C02DR0C5XUY - It was decided that this test case is not worth further implementing.
Updated by pdostal 5 months ago
- Related to action #166781: skopeo test is broken when podman is missing added