Project

General

Profile

action #101262

coordination #101048: [epic] Investigate and fix higher instability of openqaworker-arm-4/5 vs. arm-1/2/3

Document running os-autoinst full-stack.t on OSD workers size:M

Added by okurz 3 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2021-10-21
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

To verify that especially typing is stable we should be able to run the os-autoinst full-stack.t on every worker of OSD

Acceptance criteria

  • AC1: We have documentation for how to run full-stack.t on a worker
  • AC2: The test full-stack.t within the command succeeds on at least one worker host

Ideas

  • Create a script that is called after upgrade or installation as part of the os-autoinst spec file. That script would checkout the working copy of the git version linked to what was installed and run prove -I. t/full-stack.t. The script would fail if test fails and inform us, e.g. by failing the actual package installation hence our deployment would fail and we would get the message

Out of scope

  • Regular runs of the test on all workers

History

#1 Updated by okurz 3 months ago

For a start we could try to run those tests from container images so that we could run and compare the results on our openqaworker-arm machines

#2 Updated by okurz 2 months ago

  • Target version changed from future to Ready

#3 Updated by cdywan 2 months ago

  • Subject changed from Run os-autoinst full-stack.t on every worker of OSD to Document running os-autoinst full-stack.t OSD workers size:M
  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by okurz 2 months ago

  • Description updated (diff)

#5 Updated by tinita 2 months ago

  • Subject changed from Document running os-autoinst full-stack.t OSD workers size:M to Document running os-autoinst full-stack.t on OSD workers size:M

#6 Updated by okurz 2 months ago

  • Priority changed from Normal to Low

#7 Updated by okurz about 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz

This should work

command -v podman >/dev/null || zypper -n in podman
podman run --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && cd /opt/os-autoinst/ && make && prove -I. t/99-full-stack.t'

And https://github.com/os-autoinst/os-autoinst/pull/1879 should allow to simplify to

podman run --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'

however this does use qemu emulation and does not rely on KVM. So might be worth to try it out on machines but if it passes this does not guarantee that KVM behaves well. I realized that unlike in the openQA full stack test seems like we never cared to use KVM in the os-autoinst full stack test which could even be an improvement there as well.

#8 Updated by okurz about 1 month ago

Additionally https://github.com/os-autoinst/os-autoinst/pull/1884 was merged so now we can call

podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'

this looks nice in my environment. I can call this on other machines now. But also i came to the idea to have an automatic KVM detection included which would properly test hardware accelerated virtualization:

https://github.com/os-autoinst/os-autoinst/pull/1889

#9 Updated by cdywan about 1 month ago

AC2: The test full-stack.t within the command succeeds on at least one worker host

Maybe you can do that via salt, and that should cover it

#10 Updated by okurz about 1 month ago

Called

sudo salt -C 'G@roles:worker' cmd.run "zypper -n in podman && podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'

the test passed on multiple workers, e.g.:

openqaworker3.suse.de:
    Retrieving repository 'devel_openQA' metadata [..done]
    Building repository 'devel_openQA' cache [....done]
    Loading repository data...
    Reading installed packages...
    Resolving package dependencies...

    The following 3 recommended packages were automatically selected:
      cni-plugins criu podman-cni-config

    The following package is suggested, but will not be installed:
      katacontainers

    The following 14 NEW packages are going to be installed:
      catatonit cni cni-plugins conmon criu fuse-overlayfs libcontainers-common libnet9 podman podman-cni-config python3-ipaddr python3-protobuf runc slirp4netns

    14 new packages to install.
    Overall download size: 62.8 MiB. Already cached: 0 B. After the operation, additional 190.2 MiB will be used.
    Continue? [y/n/v/...? shows all options] (y): y
    Retrieving package catatonit-0.1.5-3.3.2.x86_64 (1/14), 257.2 KiB (696.5 KiB unpacked)
    Retrieving: catatonit-0.1.5-3.3.2.x86_64.rpm [done]
    Retrieving package cni-0.7.1-3.5.3.x86_64 (2/14),   2.3 MiB (  6.6 MiB unpacked)
…
    [1/2] Symlinking created executables into source directory?[K
    [1/2] cd /opt/os-autoinst && /opt/os-autoinst/tools/invoke-tests --prove-tool /usr/bin/prove --make-tool /usr/bin/ninja --build-directory /opt/os-autoinst/build?[K
    [10:18:22] t/99-full-stack.t ..
    [10:18:22] t/99-full-stack.t .. 1/?
    [10:18:22] t/99-full-stack.t .. ok   175987 ms ( 0.05 usr  0.00 sys + 52.93 cusr  5.42 csys = 58.40 CPU)
    [10:21:18]
    ?[32mAll tests successful.
    ?[0mFiles=1, Tests=25, 176 wallclock secs ( 0.06 usr  0.01 sys + 52.93 cusr  5.42 csys = 58.42 CPU)
    Result: PASS
    make: Leaving directory '/opt/os-autoinst'

others like openqaworker-arm-3 failed:

openqaworker-arm-3.suse.de:
…
    Trying to pull registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest...
…
    standard_init_linux.go:228: exec user process caused: exec format error

#11 Updated by okurz about 1 month ago

  • Status changed from In Progress to Feedback

waiting for os-autoinst README changes in https://github.com/os-autoinst/os-autoinst/pull/1890

#12 Updated by okurz about 1 month ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF