action #101262
closedcoordination #101048: [epic] Investigate and fix higher instability of openqaworker-arm-4/5 vs. arm-1/2/3
Document running os-autoinst full-stack.t on OSD workers size:M
Description
Motivation¶
To verify that especially typing is stable we should be able to run the os-autoinst full-stack.t on every worker of OSD
Acceptance criteria¶
- AC1: We have documentation for how to run full-stack.t on a worker
- AC2: The test full-stack.t within the command succeeds on at least one worker host
Ideas¶
- Create a script that is called after upgrade or installation as part of the os-autoinst spec file. That script would checkout the working copy of the git version linked to what was installed and run
prove -I. t/full-stack.t. The script would fail if test fails and inform us, e.g. by failing the actual package installation hence our deployment would fail and we would get the message
Out of scope¶
- Regular runs of the test on all workers
Updated by okurz over 2 years ago
For a start we could try to run those tests from container images so that we could run and compare the results on our openqaworker-arm machines
Updated by livdywan over 2 years ago
- Subject changed from Run os-autoinst full-stack.t on every worker of OSD to Document running os-autoinst full-stack.t OSD workers size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by tinita over 2 years ago
- Subject changed from Document running os-autoinst full-stack.t OSD workers size:M to Document running os-autoinst full-stack.t on OSD workers size:M
Updated by okurz over 2 years ago
- Status changed from Workable to In Progress
- Assignee set to okurz
This should work
command -v podman >/dev/null || zypper -n in podman
podman run --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && cd /opt/os-autoinst/ && make && prove -I. t/99-full-stack.t'
And https://github.com/os-autoinst/os-autoinst/pull/1879 should allow to simplify to
podman run --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'
however this does use qemu emulation and does not rely on KVM. So might be worth to try it out on machines but if it passes this does not guarantee that KVM behaves well. I realized that unlike in the openQA full stack test seems like we never cared to use KVM in the os-autoinst full stack test which could even be an improvement there as well.
Updated by okurz over 2 years ago
Additionally https://github.com/os-autoinst/os-autoinst/pull/1884 was merged so now we can call
podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'
this looks nice in my environment. I can call this on other machines now. But also i came to the idea to have an automatic KVM detection included which would properly test hardware accelerated virtualization:
Updated by livdywan over 2 years ago
AC2: The test full-stack.t within the command succeeds on at least one worker host
Maybe you can do that via salt, and that should cover it
Updated by okurz over 2 years ago
Called
sudo salt -C 'G@roles:worker' cmd.run "zypper -n in podman && podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'
the test passed on multiple workers, e.g.:
openqaworker3.suse.de:
Retrieving repository 'devel_openQA' metadata [..done]
Building repository 'devel_openQA' cache [....done]
Loading repository data...
Reading installed packages...
Resolving package dependencies...
The following 3 recommended packages were automatically selected:
cni-plugins criu podman-cni-config
The following package is suggested, but will not be installed:
katacontainers
The following 14 NEW packages are going to be installed:
catatonit cni cni-plugins conmon criu fuse-overlayfs libcontainers-common libnet9 podman podman-cni-config python3-ipaddr python3-protobuf runc slirp4netns
14 new packages to install.
Overall download size: 62.8 MiB. Already cached: 0 B. After the operation, additional 190.2 MiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
Retrieving package catatonit-0.1.5-3.3.2.x86_64 (1/14), 257.2 KiB (696.5 KiB unpacked)
Retrieving: catatonit-0.1.5-3.3.2.x86_64.rpm [done]
Retrieving package cni-0.7.1-3.5.3.x86_64 (2/14), 2.3 MiB ( 6.6 MiB unpacked)
…
[1/2] Symlinking created executables into source directory?[K
[1/2] cd /opt/os-autoinst && /opt/os-autoinst/tools/invoke-tests --prove-tool /usr/bin/prove --make-tool /usr/bin/ninja --build-directory /opt/os-autoinst/build?[K
[10:18:22] t/99-full-stack.t ..
[10:18:22] t/99-full-stack.t .. 1/?
[10:18:22] t/99-full-stack.t .. ok 175987 ms ( 0.05 usr 0.00 sys + 52.93 cusr 5.42 csys = 58.40 CPU)
[10:21:18]
?[32mAll tests successful.
?[0mFiles=1, Tests=25, 176 wallclock secs ( 0.06 usr 0.01 sys + 52.93 cusr 5.42 csys = 58.42 CPU)
Result: PASS
make: Leaving directory '/opt/os-autoinst'
others like openqaworker-arm-3 failed:
openqaworker-arm-3.suse.de:
…
Trying to pull registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest...
…
standard_init_linux.go:228: exec user process caused: exec format error
Updated by okurz over 2 years ago
- Status changed from In Progress to Feedback
waiting for os-autoinst README changes in https://github.com/os-autoinst/os-autoinst/pull/1890
Updated by okurz over 2 years ago
- Status changed from Feedback to Resolved
PR merged. Documentation available on https://github.com/os-autoinst/os-autoinst#verifying-a-runtime-environment