action #96383
closedDecision about where to schedule buildah and validate_btrfs tests
0%
Description
We currently have 2 types of scenarios for Container Updates/Latest:
- engines_and_tools: host features/packages (e.g. docker functinalities, podman, etc) where we also test 3rd party images on the host
- sle_image_on_sle_host: where we test all the already RELEASED SLE images on the Host under test.
The question we need to answer are:
- is
buildah_podmantest case covering a feature of the host? or it is used to validate sle images? (also applicable forbuildah_docker) - is
validate_btrfstest case covering a feature of the host? or it is used to validate sle images?
If we say they are host features, then we should schedule them only in engines_and_tools ONLY, and just pick an image to be used by the tests (e.g. opensuse or some released sle image).
Updated by jlausuch almost 3 years ago
- Related to action #95626: Setup group latest for 15sp4 added
Updated by jlausuch almost 3 years ago
I agree with @ybonatakis that validate_btrfs is a host test (testing the docker storage feature).
Updated by ybonatakis almost 3 years ago
buildah is unlikely to find any problem on the images other than podman will find. this is because it used by podman underneath. as such it is a test for host IMO and it is more likely to find bugs against the sut rather the images.
For instance all this time we caught one bug[0] and that was related to host, not the images
validate_btrfs tests the docker storage fs feature which likely belongs to the host tests as well. The image has only run btrfs as requirement IIRC
Updated by jlausuch almost 3 years ago
ybonatakis wrote:
buildah is unlikely to find any problem on the images other than podman will find. this is because it used by podman underneath. as such it is a test for host IMO and it is more likely to find bugs against the sut rather the images.
For instance all this time we caught one bug[0] and that was related to host, not the imagesvalidate_btrfs tests the docker storage fs feature which likely belongs to the host tests as well. The image has only run btrfs as requirement IIRC
Ok, then decided. Those should be run only in engines_and_tools and use a single container image (e.g. opensuse). I will prepare for it.
Updated by ybonatakis almost 3 years ago
- Status changed from New to In Progress
- Assignee set to ybonatakis
Updated by ybonatakis almost 3 years ago
i have found that jose had created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13004. however if i am not mistaken this is not what we agreed above. So i am not sure if it is old commit or something but i created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13021.
Updated by ybonatakis almost 3 years ago
- Status changed from In Progress to Feedback
Additional fix: i found that engines_and_tools_docker.yaml uses HOST_VERSION variable but it is not set in the test suite. If i am right this was intended to be a container-based variable.
Updated by jlausuch almost 3 years ago
ybonatakis wrote:
i have found that jose had created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13004. however if i am not mistaken this is not what we agreed above. So i am not sure if it is old commit or something but i created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13021.
yes, it was created before this ticket.
Updated by jlausuch almost 3 years ago
ybonatakis wrote:
Additional fix: i found that engines_and_tools_docker.yaml uses HOST_VERSION variable but it is not set in the test suite. If i am right this was intended to be a container-based variable.
HOST_VERSION is needed specially for Image-Updates because the VERSION variable refers to the Image we are testing, not the Host.
Updated by ybonatakis almost 3 years ago
jlausuch wrote:
ybonatakis wrote:
Additional fix: i found that engines_and_tools_docker.yaml uses HOST_VERSION variable but it is not set in the test suite. If i am right this was intended to be a container-based variable.
HOST_VERSION is needed specially for Image-Updates because the VERSION variable refers to the Image we are testing, not the Host.
We are saying the same thing, right? engines_and_tools_docker.yaml is not used on Image-Updates so the yaml is wrong as such
Updated by jlausuch almost 3 years ago
ybonatakis wrote:
jlausuch wrote:
ybonatakis wrote:
Additional fix: i found that engines_and_tools_docker.yaml uses HOST_VERSION variable but it is not set in the test suite. If i am right this was intended to be a container-based variable.
HOST_VERSION is needed specially for Image-Updates because the VERSION variable refers to the Image we are testing, not the Host.
We are saying the same thing, right? engines_and_tools_docker.yaml is not used on Image-Updates so the yaml is wrong as such
Yep, exactly. engines_and_tools only for host container env validation, it shouldn't be enabled for Image-Updates as the host is always the same.
Updated by ybonatakis almost 3 years ago
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13055 to adjust the validate_btrfs. it should run against unreleased images in latest group. with release to the rest. @joseph? do i miss something?
Updated by jlausuch almost 3 years ago
ybonatakis wrote:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13055 to adjust the
validate_btrfs. it should run against unreleased images in latest group. with release to the rest. @joseph? do i miss something?
Since now we have 2 groups for latest_host and latest_image, we also split the hosts tests from the image tests for 15-SP4, so validate_btrfs will stay only in latest_host.
Updated by ybonatakis almost 3 years ago
- Status changed from Feedback to Resolved
i think we can resolve this