action #96383
closed
Decision about where to schedule buildah and validate_btrfs tests
Added by jlausuch almost 3 years ago.
Updated almost 3 years ago.
Description
We currently have 2 types of scenarios for Container Updates/Latest:
- engines_and_tools: host features/packages (e.g. docker functinalities, podman, etc) where we also test 3rd party images on the host
- sle_image_on_sle_host: where we test all the already RELEASED SLE images on the Host under test.
The question we need to answer are:
- is
buildah_podman test case covering a feature of the host? or it is used to validate sle images? (also applicable for buildah_docker)
- is
validate_btrfs test case covering a feature of the host? or it is used to validate sle images?
If we say they are host features, then we should schedule them only in engines_and_tools ONLY, and just pick an image to be used by the tests (e.g. opensuse or some released sle image).
I agree with @ybonatakis that validate_btrfs is a host test (testing the docker storage feature).
buildah is unlikely to find any problem on the images other than podman will find. this is because it used by podman underneath. as such it is a test for host IMO and it is more likely to find bugs against the sut rather the images.
For instance all this time we caught one bug[0] and that was related to host, not the images
validate_btrfs tests the docker storage fs feature which likely belongs to the host tests as well. The image has only run btrfs as requirement IIRC
[0] https://bugzilla.suse.com/show_bug.cgi?id=1185143
ybonatakis wrote:
buildah is unlikely to find any problem on the images other than podman will find. this is because it used by podman underneath. as such it is a test for host IMO and it is more likely to find bugs against the sut rather the images.
For instance all this time we caught one bug[0] and that was related to host, not the images
validate_btrfs tests the docker storage fs feature which likely belongs to the host tests as well. The image has only run btrfs as requirement IIRC
[0] https://bugzilla.suse.com/show_bug.cgi?id=1185143
Ok, then decided. Those should be run only in engines_and_tools and use a single container image (e.g. opensuse). I will prepare for it.
- Status changed from New to In Progress
- Assignee set to ybonatakis
- Status changed from In Progress to Feedback
Additional fix: i found that engines_and_tools_docker.yaml uses HOST_VERSION variable but it is not set in the test suite. If i am right this was intended to be a container-based variable.
ybonatakis wrote:
Additional fix: i found that engines_and_tools_docker.yaml uses HOST_VERSION variable but it is not set in the test suite. If i am right this was intended to be a container-based variable.
HOST_VERSION is needed specially for Image-Updates because the VERSION variable refers to the Image we are testing, not the Host.
jlausuch wrote:
ybonatakis wrote:
Additional fix: i found that engines_and_tools_docker.yaml uses HOST_VERSION variable but it is not set in the test suite. If i am right this was intended to be a container-based variable.
HOST_VERSION is needed specially for Image-Updates because the VERSION variable refers to the Image we are testing, not the Host.
We are saying the same thing, right? engines_and_tools_docker.yaml is not used on Image-Updates so the yaml is wrong as such
ybonatakis wrote:
jlausuch wrote:
ybonatakis wrote:
Additional fix: i found that engines_and_tools_docker.yaml uses HOST_VERSION variable but it is not set in the test suite. If i am right this was intended to be a container-based variable.
HOST_VERSION is needed specially for Image-Updates because the VERSION variable refers to the Image we are testing, not the Host.
We are saying the same thing, right? engines_and_tools_docker.yaml is not used on Image-Updates so the yaml is wrong as such
Yep, exactly. engines_and_tools only for host container env validation, it shouldn't be enabled for Image-Updates as the host is always the same.
ybonatakis wrote:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13055 to adjust the validate_btrfs. it should run against unreleased images in latest group. with release to the rest. @joseph? do i miss something?
Since now we have 2 groups for latest_host and latest_image, we also split the hosts tests from the image tests for 15-SP4, so validate_btrfs will stay only in latest_host.
- Status changed from Feedback to Resolved
i think we can resolve this
Also available in: Atom
PDF