coordination #101048: [epic] Investigate and fix higher instability of openqaworker-arm-4/5 vs. arm-1/2/3
Document running os-autoinst full-stack.t on OSD workers size:M
To verify that especially typing is stable we should be able to run the os-autoinst full-stack.t on every worker of OSD
- AC1: We have documentation for how to run full-stack.t on a worker
- AC2: The test full-stack.t within the command succeeds on at least one worker host
- Create a script that is called after upgrade or installation as part of the os-autoinst spec file. That script would checkout the working copy of the git version linked to what was installed and run
prove -I. t/full-stack.t. The script would fail if test fails and inform us, e.g. by failing the actual package installation hence our deployment would fail and we would get the message
Out of scope¶
- Regular runs of the test on all workers
#7 Updated by okurz about 2 months ago
- Status changed from Workable to In Progress
- Assignee set to okurz
This should work
command -v podman >/dev/null || zypper -n in podman podman run --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && cd /opt/os-autoinst/ && make && prove -I. t/99-full-stack.t'
And https://github.com/os-autoinst/os-autoinst/pull/1879 should allow to simplify to
podman run --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'
however this does use qemu emulation and does not rely on KVM. So might be worth to try it out on machines but if it passes this does not guarantee that KVM behaves well. I realized that unlike in the openQA full stack test seems like we never cared to use KVM in the os-autoinst full stack test which could even be an improvement there as well.
#8 Updated by okurz about 1 month ago
Additionally https://github.com/os-autoinst/os-autoinst/pull/1884 was merged so now we can call
podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'
this looks nice in my environment. I can call this on other machines now. But also i came to the idea to have an automatic KVM detection included which would properly test hardware accelerated virtualization:
#10 Updated by okurz about 1 month ago
sudo salt -C 'G@roles:worker' cmd.run "zypper -n in podman && podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'
the test passed on multiple workers, e.g.:
openqaworker3.suse.de: Retrieving repository 'devel_openQA' metadata [..done] Building repository 'devel_openQA' cache [....done] Loading repository data... Reading installed packages... Resolving package dependencies... The following 3 recommended packages were automatically selected: cni-plugins criu podman-cni-config The following package is suggested, but will not be installed: katacontainers The following 14 NEW packages are going to be installed: catatonit cni cni-plugins conmon criu fuse-overlayfs libcontainers-common libnet9 podman podman-cni-config python3-ipaddr python3-protobuf runc slirp4netns 14 new packages to install. Overall download size: 62.8 MiB. Already cached: 0 B. After the operation, additional 190.2 MiB will be used. Continue? [y/n/v/...? shows all options] (y): y Retrieving package catatonit-0.1.5-3.3.2.x86_64 (1/14), 257.2 KiB (696.5 KiB unpacked) Retrieving: catatonit-0.1.5-3.3.2.x86_64.rpm [done] Retrieving package cni-0.7.1-3.5.3.x86_64 (2/14), 2.3 MiB ( 6.6 MiB unpacked) … [1/2] Symlinking created executables into source directory?[K [1/2] cd /opt/os-autoinst && /opt/os-autoinst/tools/invoke-tests --prove-tool /usr/bin/prove --make-tool /usr/bin/ninja --build-directory /opt/os-autoinst/build?[K [10:18:22] t/99-full-stack.t .. [10:18:22] t/99-full-stack.t .. 1/? [10:18:22] t/99-full-stack.t .. ok 175987 ms ( 0.05 usr 0.00 sys + 52.93 cusr 5.42 csys = 58.40 CPU) [10:21:18] ?[32mAll tests successful. ?[0mFiles=1, Tests=25, 176 wallclock secs ( 0.06 usr 0.01 sys + 52.93 cusr 5.42 csys = 58.42 CPU) Result: PASS make: Leaving directory '/opt/os-autoinst'
others like openqaworker-arm-3 failed:
openqaworker-arm-3.suse.de: … Trying to pull registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest... … standard_init_linux.go:228: exec user process caused: exec format error
#11 Updated by okurz about 1 month ago
- Status changed from In Progress to Feedback
waiting for os-autoinst README changes in https://github.com/os-autoinst/os-autoinst/pull/1890
#12 Updated by okurz about 1 month ago
- Status changed from Feedback to Resolved
PR merged. Documentation available on https://github.com/os-autoinst/os-autoinst#verifying-a-runtime-environment