Project

General

Profile

Actions

action #163553

closed

Add API compatability test to skopeo

Added by ph03nix 7 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
2024-07-09
Due date:
% Done:

0%

Estimated time:

Description

The current skopeo test runs test only skopeo in it's default configuration. https://bugzilla.suse.com/show_bug.cgi?id=1219563 shows that we should also test skopeo in conjunction with docker to detect such issues in the test runs.

This ticket is about extending the current skopeo test runs in such a way, that it can run with the podman (should remain the default) and with the docker daemon.

Acceptance criteria

  • Modify skopeo.pm such, that it uses the docker daemon for docker test runs
  • Add the skopeo tests to the docker test runs

Related issues 2 (0 open2 closed)

Related to Containers and images - action #154933: Create test for skopeoResolvedetchubykalo2024-02-05

Actions
Related to Containers and images - action #166781: skopeo test is broken when podman is missingResolvedhilchev2024-09-13

Actions
Actions #1

Updated by ph03nix 7 months ago

Actions #2

Updated by hilchev 6 months ago

Alright, so I took a look at the current situation.

  1. We have a Skopeo test module that is very runtime-agnostic. It should be able to run just fine on docker or on podman. https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/containers/skopeo.pm
  2. We have the CONTAINER_RUNTIMES variable, which can take multiple values - e.g. podman,docker
  3. I only see the Skopeo test being loaded here: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/0faff221ab2bc5a8d56dea28c550c72ea8536a29/lib/main_containers.pm#L145

So now I'm thinking about the implementation - the obvious way seems to be to make sure that passing both runtimes in the variables spins up two test runs, one for each runtime. I'm not sure how much effort that would be so I'm digging in to figure it out. Alternatively, a change in the codebase would make it so that we ALWAYS test on both runtimes, but that seems a bit hardcode-y

Actions #3

Updated by hilchev 6 months ago

I scoured O3 for test cases but I couldn't find anything. Thankfully Martin Loviska sent me the exact test run I needed (and was kind enough to discuss my potential solution) : https://openqa.opensuse.org/tests/4397144# . So passing both values to the CONTAINER_RUNTIMES does indeed results in two executions.

These two executions however seem moot. Skopeo does not rely on any container runtime or daemon. Both inspect and copy seem to work fine without anything else installed.

In that case, the issue becomes a matter of "Do we test against newer Registry versions?". I went back to the Bugzilla and read it more thoroughly and from what I understand, the original issue was the packages in the OS itself were stale.

TL;DR; If we make sure we are running the Skopeo test against an Image Registry that is known to have the latest Registry API, the test is good enough.

Actions #4

Updated by hilchev 6 months ago

After some further testing, I think the potential improvements to the test module would be:

  1. Test against more than one registry (e.g. inspect something from Quay.io or Dockerhub.io.). Have to figure out how to get around the pull limit from public registries if we use those.
  2. Spin up a local docker registry and test against it.
  3. Get info on the registry we're using's version. There is not straightforward way to do that afaik.
Actions #5

Updated by ph03nix 6 months ago

  • Subject changed from Create docker/skopeo test runs to Create docker+skopeo test runs
Actions #6

Updated by ph03nix 6 months ago

We had a clarification meeting with the following outcome:

  1. The issue in https://bugzilla.suse.com/show_bug.cgi?id=1219563 was that there a too old version of skopeo was incompatible with the API of a remote registry (unknown origin, could be docker.io, registry.suse.com, registry.opensuse.org, quay.io, ghcr.io, ...)
  2. We can partly test for such issues by spinning up a own suse-registry container and test skopeo against that one
  3. For point number 2 it is ok to modify skopeo.pm in such a way, that it required podman to be installed. This includes unscheduling skopeo.pm for any non-podman test runs.
Actions #7

Updated by ph03nix 6 months ago

  • Subject changed from Create docker+skopeo test runs to Add API compatability test to skopeo
Actions #8

Updated by ph03nix 6 months ago ยท Edited

After reading https://bugzilla.suse.com/show_bug.cgi?id=1219563#c21, it appears to me that the issue was the interaction between docker and buildah, and skopeo is unrelated:


15sp5docker:~ # docker run -it  --privileged -v /var/run/docker.sock:/var/run/docker.sock registry.suse.com/bci/bci-base:15.5
1d0a71969a4b:/ # zypper in -y skopeo buildah
1d0a71969a4b:/ # rpm -q skopeo buildah
skopeo-1.14.4-150300.11.11.1.x86_64
buildah-1.34.1-150500.3.7.1.x86_64
1d0a71969a4b:/ # sed -i 's/^driver =.*/driver="vfs"/' /etc/containers/storage.conf
1d0a71969a4b:/ # buildah from  docker-daemon:registry.suse.com/bci/bci-base:15.5
Error: initializing source docker-daemon:registry.suse.com/bci/bci-base:15.5: loading image from docker engine: Error response from daemon: client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

I'm not sure if this is something we should actually test, will ask for clarification: https://suse.slack.com/archives/C02DR0C5XUY/p1723723540616939.

Actions #10

Updated by ph03nix 5 months ago

  • Status changed from Workable to Closed

See https://suse.slack.com/archives/C02DR0C5XUY/p1725021286474529?thread_ts=1723723540.616939&cid=C02DR0C5XUY - It was decided that this test case is not worth further implementing.

Actions #11

Updated by pdostal 5 months ago

  • Related to action #166781: skopeo test is broken when podman is missing added
Actions #12

Updated by ph03nix 4 months ago

  • Tags set to containers
Actions

Also available in: Atom PDF