Project

General

Profile

Actions
Copy link

action #101262

closed

coordination #101048: [epic] Investigate and fix higher instability of openqaworker-arm-4/5 vs. arm-1/2/3

Document running os-autoinst full-stack.t on OSD workers size:M

Added by okurz about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Low
Assignee:
okurz
Category:
Feature requests
Target version:
Ready
Start date:
2021-10-21
Due date:
% Done:

0%

Estimated time:

Description

Motivation

To verify that especially typing is stable we should be able to run the os-autoinst full-stack.t on every worker of OSD

Acceptance criteria

  • AC1: We have documentation for how to run full-stack.t on a worker
  • AC2: The test full-stack.t within the command succeeds on at least one worker host

Ideas

  • Create a script that is called after upgrade or installation as part of the os-autoinst spec file. That script would checkout the working copy of the git version linked to what was installed and run prove -I. t/full-stack.t. The script would fail if test fails and inform us, e.g. by failing the actual package installation hence our deployment would fail and we would get the message

Out of scope

  • Regular runs of the test on all workers
Actions
Copy link
 #1

Updated by okurz about 3 years ago

For a start we could try to run those tests from container images so that we could run and compare the results on our openqaworker-arm machines

Actions
Copy link
 #2

Updated by okurz about 3 years ago

  • Target version changed from future to Ready
Actions
Copy link
 #3

Updated by livdywan about 3 years ago

  • Subject changed from Run os-autoinst full-stack.t on every worker of OSD to Document running os-autoinst full-stack.t OSD workers size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #4

Updated by okurz about 3 years ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by tinita about 3 years ago

  • Subject changed from Document running os-autoinst full-stack.t OSD workers size:M to Document running os-autoinst full-stack.t on OSD workers size:M
Actions
Copy link
 #6

Updated by okurz about 3 years ago

  • Priority changed from Normal to Low
Actions
Copy link
 #7

Updated by okurz about 3 years ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz

This should work

command -v podman >/dev/null || zypper -n in podman
podman run --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && cd /opt/os-autoinst/ && make && prove -I. t/99-full-stack.t'

And https://github.com/os-autoinst/os-autoinst/pull/1879 should allow to simplify to

podman run --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'

however this does use qemu emulation and does not rely on KVM. So might be worth to try it out on machines but if it passes this does not guarantee that KVM behaves well. I realized that unlike in the openQA full stack test seems like we never cared to use KVM in the os-autoinst full stack test which could even be an improvement there as well.

Actions
Copy link
 #8

Updated by okurz almost 3 years ago

Additionally https://github.com/os-autoinst/os-autoinst/pull/1884 was merged so now we can call

podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'

this looks nice in my environment. I can call this on other machines now. But also i came to the idea to have an automatic KVM detection included which would properly test hardware accelerated virtualization:

https://github.com/os-autoinst/os-autoinst/pull/1889

Actions
Copy link
 #9

Updated by livdywan almost 3 years ago

AC2: The test full-stack.t within the command succeeds on at least one worker host

Maybe you can do that via salt, and that should cover it

Actions
Copy link
 #10

Updated by okurz almost 3 years ago

Called

sudo salt -C 'G@roles:worker' cmd.run "zypper -n in podman && podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ symlinks test-perl-testsuite TESTS=t/99-full-stack.t'

the test passed on multiple workers, e.g.:

openqaworker3.suse.de:
    Retrieving repository 'devel_openQA' metadata [..done]
    Building repository 'devel_openQA' cache [....done]
    Loading repository data...
    Reading installed packages...
    Resolving package dependencies...

    The following 3 recommended packages were automatically selected:
      cni-plugins criu podman-cni-config

    The following package is suggested, but will not be installed:
      katacontainers

    The following 14 NEW packages are going to be installed:
      catatonit cni cni-plugins conmon criu fuse-overlayfs libcontainers-common libnet9 podman podman-cni-config python3-ipaddr python3-protobuf runc slirp4netns

    14 new packages to install.
    Overall download size: 62.8 MiB. Already cached: 0 B. After the operation, additional 190.2 MiB will be used.
    Continue? [y/n/v/...? shows all options] (y): y
    Retrieving package catatonit-0.1.5-3.3.2.x86_64 (1/14), 257.2 KiB (696.5 KiB unpacked)
    Retrieving: catatonit-0.1.5-3.3.2.x86_64.rpm [done]
    Retrieving package cni-0.7.1-3.5.3.x86_64 (2/14),   2.3 MiB (  6.6 MiB unpacked)
…
    [1/2] Symlinking created executables into source directory?[K
    [1/2] cd /opt/os-autoinst && /opt/os-autoinst/tools/invoke-tests --prove-tool /usr/bin/prove --make-tool /usr/bin/ninja --build-directory /opt/os-autoinst/build?[K
    [10:18:22] t/99-full-stack.t ..
    [10:18:22] t/99-full-stack.t .. 1/?
    [10:18:22] t/99-full-stack.t .. ok   175987 ms ( 0.05 usr  0.00 sys + 52.93 cusr  5.42 csys = 58.40 CPU)
    [10:21:18]
    ?[32mAll tests successful.
    ?[0mFiles=1, Tests=25, 176 wallclock secs ( 0.06 usr  0.01 sys + 52.93 cusr  5.42 csys = 58.42 CPU)
    Result: PASS
    make: Leaving directory '/opt/os-autoinst'

others like openqaworker-arm-3 failed:

openqaworker-arm-3.suse.de:
…
    Trying to pull registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest...
…
    standard_init_linux.go:228: exec user process caused: exec format error
Actions
Copy link
 #11

Updated by okurz almost 3 years ago

  • Status changed from In Progress to Feedback

waiting for os-autoinst README changes in https://github.com/os-autoinst/os-autoinst/pull/1890

Actions
Copy link
 #12

Updated by okurz almost 3 years ago

  • Status changed from Feedback to Resolved
Actions
Copy link

Also available in: Atom PDF