QA » openQA Project

both PRs merged

deployed on o3.

Created
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6527
to check

Idea: Use "mentions" in PR description or git commits to instruct openQA to test certain branches, e.g. @openqa: Clone https://openqa.opensuse.org/tests/828636 could mean what I will execute manually now:

HOST=https://openqa.opensuse.org
JOB=828636
TESTSUITE=kubeadm-retest
REPO_NAME=okurz/os-autoinst-distri-opensuse
REPO="https://github.com/$REPO_NAME.git"
BRANCH=fix/kubic_repositories
REPO_BRANCH="$REPO_NAME#$BRANCH"
TEST="$TESTSUITE@$REPO_BRANCH"
PR=6527
BUILD="$REPO_NAME#$PR"
CASEDIR="$REPO#$BRANCH"
openqa-clone-job --skip-download --skip-chained-deps --from $HOST --host $HOST $JOB _GROUP=0 TEST=$TEST BUILD=$BUILD CASEDIR=$CASEDIR

-> https://openqa.opensuse.org/tests/829414/file/vars.json

looks like caching overwrote CASEDIR again.

Of course, I updated the web UI after I merged the openQA PR but I need the change active in the package "openQA-worker" on the workers instead…

ok, power8 hasn't been updated since 2018-11 anyway so I will just update this worker and try there:

HOST=https://openqa.opensuse.org
JOB=828144
TESTSUITE=extra_tests_in_textmode
REPO_NAME=okurz/os-autoinst-distri-opensuse
REPO="https://github.com/$REPO_NAME.git"
BRANCH=feature/custom_git_branch
REPO_BRANCH="$REPO_NAME#$BRANCH"
TEST="$TESTSUITE@$REPO_BRANCH"
PR=6529
BUILD="$REPO_NAME#$PR"
CASEDIR="$REPO#$BRANCH"
openqa-clone-job --skip-download --skip-chained-deps --from $HOST --host $HOST $JOB _GROUP=0 TEST=$TEST BUILD=$BUILD CASEDIR=$CASEDIR

-> https://openqa.opensuse.org/tests/829419#step/boot_to_desktop/1 works when I have disabled the apparmor profile because of /usr/bin/env denied

-> https://github.com/os-autoinst/openQA/pull/1952

apparmor changes should be deployed to all workers, updated power8 again manually. https://openqa.opensuse.org/tests/831741 triggered to check current state.

Failed because of further missing parts.

type=AVC msg=audit(1547567641.923:290519): apparmor="DENIED" operation="open" profile="/usr/share/openqa/script/worker" name="/usr/share/git-core/templates/" pid=140378 comm="git" requested_mask="r" denied_mask="r" fsuid=488 ouid=0
type=AVC msg=audit(1547567641.953:290520): apparmor="DENIED" operation="exec" profile="/usr/share/openqa/script/worker" name="/usr/lib/git/git-remote-http" pid=140379 comm="git" requested_mask="x" denied_mask="x" fsuid=488 ouid=0

Setting profile to complain on power8 and retriggering test.

-> https://github.com/os-autoinst/openQA/pull/1955

EDIT: merged and deployed manually, enforced the apparmor profile again.

hm, I triggered https://openqa.opensuse.org/t833099 but now caching overwrites the PRODUCTDIR. Triggered https://openqa.opensuse.org/t833355 with PRODUCTDIR=products/opensuse but then the path is also not found. I would need to directly reference the path from the pool dir I assume so https://openqa.opensuse.org/t833376 with PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse does not find needles because os-autoinst-distri-opensuse/products/opensuse/needles does not exist. So for this I should explicitly set NEEDLESDIR but to the absolute path pointing to the cached copy. (Seems like it would have been easier to do the git handling in openQA but that's the drawback of choosing letting os-autoinst support it already on the low level). So

PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse
NEEDLES_DIR=/var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles
openqa-clone-job --skip-download --skip-chained-deps --from $HOST --host $HOST $JOB _GROUP=0 TEST=$TEST BUILD=$BUILD CASEDIR=$CASEDIR PRODUCTDIR=$PRODUCTDIR NEEDLES_DIR=$NEEDLES_DIR

-> https://openqa.opensuse.org/t833378 which worked.

The job in #44327#note-4 also "worked" but PRODUCTDIR was still taken from the master's branch distribution. It depends on what we want to replace and what not how we need to setup the variables. At least we have it fully controllable and also the NEEDLES_DIR content can be dynamically put together based on variables that are defined in the clonee job.

fvogt tried it and had problems, I as well. First I created a wrapper script now to have something more reproducible. Then I adjusted also the test plan in my PR to be sure that also the test plan is really impacted:

$ bash <(curl -s https://raw.githubusercontent.com/okurz/scripts/master/openqa-clone-custom-git-refspec) https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6529 https://openqa.opensuse.org/tests/835060

Created job #836481: opensuse-Tumbleweed-DVD-ppc64le-Build20190110-extra_tests_in_textmode@ppc64le -> https://openqa.opensuse.org/t836482

Works just fine, it also loads really the schedule from the PR. Why did it fail on other workers? Is "power8" again somewhat special?

Trying on aarch64 from the latest extra_tests_in_textmode:

$ bash <(curl -s https://raw.githubusercontent.com/okurz/scripts/master/openqa-clone-custom-git-refspec) https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6529 https://openqa.opensuse.org/tests/834782

Created job #836483: opensuse-Tumbleweed-DVD-aarch64-Build20190115-extra_tests_in_textmode@aarch64 -> https://openqa.opensuse.org/t836483

Hm, failed as well as on x86_64 with

[2019-01-23T22:30:00.533 CET] [debug] git hash in /var/lib/openqa/pool/4/os-autoinst-distri-opensuse: 222117de4587e6e16823500e1ef584332e69b828
Can't locate os-autoinst-distri-opensuse/products/opensuse/main.pm in @INC (@INC contains: /var/lib/openqa/pool/4/os-autoinst-distri-opensuse/lib /var/lib/openqa/pool/4/blib/arch /var/lib/openqa/pool/4/blib/lib /usr/lib/os-autoinst /usr/lib/perl5/site_perl/5.26.1/aarch64-linux-thread-multi /usr/lib/perl5/site_perl/5.26.1 /usr/lib/perl5/vendor_perl/5.26.1/aarch64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.26.1 /usr/lib/perl5/5.26.1/aarch64-linux-thread-multi /usr/lib/perl5/5.26.1 /usr/lib/perl5/site_perl) at /usr/bin/isotovideo line 234.

so the PRODUCTDIR can not be resolved except on power8.

Trying with productdir=../products/opensuse because isotovideo adds the "lib" subfolder to the include path and I thought maybe then we can resolve from there does not work because then the -e check can not find the main.pm from the pool directory. So I guess I need to find out why on power8 we can automatically include the .pm file or I change isotovideo to explicitly add the casedir to INC or I change the file existance check. Maybe it helps if I understand first the @INC stuff better.

Now the apparmor profile seems to be off again.

https://openqa.opensuse.org/tests/838044/file/autoinst-log.txt shows "fatal: cannot opendir '/usr/share/git-core/templates/branches': Permission denied".

From openqaworker4:

type=AVC msg=audit(1548421917.020:193): apparmor="DENIED" operation="open" profile="/usr/share/openqa/script/worker" name="/usr/share/git-core/templates/branches/" pid=24572 comm="git" requested_mask="r" denied_mask="r" fsuid=471 ouid=0

openqaworker4:~ # grep templates /var/log/audit/audit.log  | tail -n 1
type=AVC msg=audit(1548421917.020:193): apparmor="DENIED" operation="open" profile="/usr/share/openqa/script/worker" name="/usr/share/git-core/templates/branches/" pid=24572 comm="git" requested_mask="r" denied_mask="r" fsuid=471 ouid=0
openqaworker4:~ # date --date='@1548421917.020'
Fri Jan 25 14:11:57 CET 2019

so I guess I should have used ** all along.

Hotpatch:

for i in power8 aarch64 imagetester openqaworker1 openqaworker4; do echo $i && ssh root@$i "sed -i -e '/templates/s@templates/\*@&*@' /etc/apparmor.d/usr.share.openqa.script.worker && systemctl reload apparmor"; done

Fix: https://github.com/os-autoinst/openQA/pull/1975

Lately again my tests are fine to at least override the test code when no schedule changes are needed. Created https://github.com/os-autoinst/openQA/pull/1980 to add a convenience script to openQA and documentation for this part. I would still like to go further to cover overriding the PRODUCTDIR properly as well.

Another idea regarding PRODUCTDIR override would be to just add support for relative paths and read starting from CASEDIR or not?

Tried to use openqa-clone-custom-git-refspec in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6728 and hit quite some problems.

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 844490 _GROUP=0 TEST=zdup-Leap-42.3-kde@okurz/os-autoinst-distri-opensuse#fix/password_prompt BUILD=okurz/os-autoinst-distri-opensuse#6728 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/password_prompt

fails in https://openqa.opensuse.org/tests/848265 as it seems my changes to lib/x11utils.pm were not effective. I tried to call openqa-clone-custom-git-refspec as well but that failed on me because of the product dir not found. I guess without overwriting the PRODUCTDIR the changes from lib-directory will not be in effect.

So let's be specific about the schedule:

openqa-clone-custom-git-refspec https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6728 https://openqa.opensuse.org/tests/847345 SCHEDULE=tests/boot/boot_to_desktop,tests/x11/user_gui_login

Created job #848301: opensuse-15.1-DVD-x86_64-Build406.2-extra_tests_on_kde@64bit -> https://openqa.opensuse.org/t848301

but that one also incompleted in https://openqa.opensuse.org/tests/848301/file/autoinst-log.txt not being able to find the main.pm file now.

Can I debug what the include path contains on different workers maybe?:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de 2440314 _GROUP=0 TEST=okurz_poo#44327 BUILD=okurz_poo#44327 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/clone_git PRODUCTDIR=os-autoinst-distri-opensuse/products/sle SCHEDULE=tests/debug_inc

Created job #2442034: sle-15-SP1-Installer-DVD-x86_64-Build159.1-memtest@64bit -> https://openqa.suse.de/t2442034

fails because "env" is not permitted. Of course osd is missing the latest apparmor adaptions :(

so let's pick a more permissive worker, openqaworker2 should work because we run all the remote backend stuff like s390x-zVM so:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de 2440314 _GROUP=0 TEST=okurz_poo#44327 BUILD=okurz_poo#44327 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/clone_git PRODUCTDIR=os-autoinst-distri-opensuse/products/sle SCHEDULE=tests/debug_inc WORKER_CLASS=s390x-zVM-vswitch-l2

-> Created job #2442036: sle-15-SP1-Installer-DVD-x86_64-Build159.1-memtest@64bit -> https://openqa.suse.de/t2442036

It "incompleted" but that is fine because I just wanted to evaluate one line to print "@INC" and I have that:

INC: /var/lib/openqa/pool/4/os-autoinst-distri-opensuse/tests /var/lib/openqa/pool/4/os-autoinst-distri-opensuse/lib /var/lib/openqa/pool/4/blib/arch /var/lib/openqa/pool/4/blib/lib /usr/lib/os-autoinst /usr/lib/perl5/site_perl/5.18.2/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.18.2 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.18.2 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi /usr/lib/perl5/5.18.2 /usr/lib/perl5/site_perl .

so the pool directory at least is not directly included. Updated the fake test module to be a complete test module, retriggered test fails because needle dir can not be found as expected so setting the same as openqa-clone-custom-git-refspec would do:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de 2440314 _GROUP=0 TEST=okurz_poo#44327 BUILD=okurz_poo#44327 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/clone_git PRODUCTDIR=os-autoinst-distri-opensuse/products/sle SCHEDULE=tests/debug_inc WORKER_CLASS=s390x-zVM-vswitch-l2 NEEDLES_DIR=/var/lib/openqa/cache/openqa.suse.de/tests/sle/products/sle/needles

-> Created job #2442045: sle-15-SP1-Installer-DVD-x86_64-Build159.1-memtest@64bit -> https://openqa.suse.de/t2442045

and that one is just fine. The exact git commit used was https://github.com/okurz/os-autoinst-distri-opensuse/commit/8e54b5e26ad85df8f6e0b9846a366ee36ee537c7

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 847590 _GROUP=0 TEST=okurz_poo#44327 BUILD=okurz_poo#44327 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/clone_git PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse SCHEDULE=tests/debug_inc NEEDLES_DIR=/var/lib/openqa/cache/openqa1-suse/tests/opensuse/products/opensuse/needles

-> Created job #848320: opensuse-Tumbleweed-DVD-x86_64-Build20190205-mediacheck@64bit -> https://openqa.opensuse.org/t848320

and on the two other classes "qemu_aarch64" and "qemu_ppc64le"

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 847590 _GROUP=0 TEST=okurz_poo#44327 BUILD=okurz_poo#44327 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/clone_git PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse SCHEDULE=tests/debug_inc NEEDLES_DIR=/var/lib/openqa/cache/openqa1-suse/tests/opensuse/products/opensuse/needles WORKER_CLASS=qemu_aarch64

Created job #848321: opensuse-Tumbleweed-DVD-x86_64-Build20190205-mediacheck@64bit -> https://openqa.opensuse.org/t848321

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 847590 _GROUP=0 TEST=okurz_poo#44327 BUILD=okurz_poo#44327 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/clone_git PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse SCHEDULE=tests/debug_inc NEEDLES_DIR=/var/lib/openqa/cache/openqa1-suse/tests/opensuse/products/opensuse/needles WORKER_CLASS=qemu_ppc64le

Created job #848322: opensuse-Tumbleweed-DVD-x86_64-Build20190205-mediacheck@64bit -> https://openqa.opensuse.org/t848322

The latter fails because it seems the hostname pointing to o3 is different, "openqa1-opensuse" instead of "openqa1-suse":

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org 847590 _GROUP=0 TEST=okurz_poo#44327 BUILD=okurz_poo#44327 CASEDIR=https://github.com/okurz/os-autoinst-distri-opensuse.git#fix/clone_git PRODUCTDIR=os-autoinst-distri-opensuse/products/opensuse SCHEDULE=tests/debug_inc NEEDLES_DIR=/var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles WORKER_CLASS=qemu_ppc64le

Created job #848324: opensuse-Tumbleweed-DVD-x86_64-Build20190205-mediacheck@64bit -> https://openqa.opensuse.org/t848324

so the osd workers seem to work, as well as power8 as reported above but not the x86_64.o.o workers and not aarch64.o.o as well

Oh geez, I think I got the difference now. OSD workers as well as power8.o.o run openSUSE Leap 42.3 / SLE12SP3 vs. Leap 15.0 so perl5.18 vs. 5.26. I guess it's https://www.effectiveperlprogramming.com/2017/01/v5-26-removes-dot-from-inc/

https://github.com/os-autoinst/os-autoinst/pull/1101

was merged, looking fine. All good on o3 and osd workers.

Had a nice discussion with trenn: https://github.com/cobbler/cobbler/pull/2024 is a nice example of what one would desire to do in openQA instead, e.g.

1. pull request in github triggers openQA test
2. openQA test executes a custom test defined in the github source repo of the product under test (cobbler in this case)
3. openQA test only boots a VM image, e.g. Tumbleweed, and executes the custom module
4. test result is fed back to the github PR

implementation suggestions:

1. could be done with polling bot for now
2. using latest os-autoinst-distri-opensuse plus – as custom assets currently do not work – a custom test module that same what we tried already for HPC gets the defined test script, could be anything downloadable and executable, and runs it
3. use SCHEDULE parameter with e.g. SCHEDULE=tests/boot/boot_to_desktop,tests/run_custom
4. github API access relying on test variables previously provided on trigger

could be that dynamic needle reloading does not work with custom needles dir, e.g. in https://openqa.suse.de/tests/2496374# . Should crosscheck that.