QA » openQA Project

merged, was automatically deployed to o3, now enabled again in /etc/openqa/openqa.ini with:

job_done_hook_failed = env scheme=http exclude_group_regex='(Development|Open Build Service|Others).*/.*(Kernel|Development|Staging|MicroOS)' /opt/os-autoinst-scripts/openqa-label-known-issues-and-investigate-hook

Updated apparmor with aa-logprof. Found the problem that openqa-cli is by default called with https causing "Connection refused" messages. I fixed that with https://github.com/os-autoinst/scripts/pull/61 and locally on o3 already. TODO: After merge of PR check that clean state is checked out in /opt/os-autoinst-scripts. In journal I still see Dec 12 11:04:39 ariel openqa-gru[1606]: /opt/os-autoinst-scripts/openqa-investigate: line 100: /usr/bin/openqa-cli: Permission denied. Apparently I missed something in apparmor. The latest change in aa-logprof suggested to include #include <abstractions/openssl> in the apparmor profile as well. But still I see:

Dec 12 15:04:44 ariel openqa-gru[1606]: /opt/os-autoinst-scripts/openqa-investigate: line 100: /usr/bin/openqa-cli: Permission denied
Dec 12 15:04:44 ariel openqa-gru[1606]: unable to query job data for 1504727:

With this no jobs are really triggered as investigation jobs. So trigger that intentionally now. Based on opensuse-Tumbleweed-NET-aarch64-textmode@aarch64 :

openqa_clone_job_o3 --skip-chained-deps 1503691 INCLUDE_MODULES=isosize ISO_MAXSIZE=42 TEST=okurz_poo80830 _GROUP=0 BUILD= _SKIP_POST_FAIL_HOOKS=1

Created job #1504805: opensuse-Tumbleweed-NET-aarch64-Build20201211-textmode@aarch64 -> https://openqa.opensuse.org/t1504805

This should trigger an early failure in "isosize" as only module.

This triggered again the error after about 30s of test execution. Trying with aa-disable /etc/apparmor.d/usr.share.openqa.script.openqa

This worked now: https://openqa.opensuse.org/tests/1504807#comment-100753 but there had been error messages:

Dec 12 22:10:35 ariel openqa-gru[1606]: jq: error (at <stdin>:1): Cannot index string with string "text"
Dec 12 22:10:35 ariel openqa-gru[1606]: parse error: Invalid numeric literal at line 1, column 10

updated https://github.com/os-autoinst/scripts/pull/61 accordingly and applied the patch on o3 temporarily.

Enforced apparmor again. Triggering again a failed job triggered an investigation job just fine, no entry in /var/log/audit/audit.log, no error entry in journal.

Time to wait for review of my PR and see how it behaves over time.

no review of https://github.com/os-autoinst/scripts/pull/61 for 3 days, self-merged now.

Did on o3 sudo -u geekotest git -C /opt/os-autoinst-scripts reset --hard origin/master to get rid of the local patch.

With the SQL query on o3 select jobs.id, result_dir from comments,jobs,users where comments.user_id = users.id and comments.job_id = jobs.id and nickname ~ 'geekotest' and text ~ 'Automatic investigation jobs'; I could find that the latest job where automatic investigation jobs were triggered was 2020-12-1 01:13:49 +0000, but "auto-review" from the gitlab CI pipeline did that as recently as 2020-12-15 so something broke maybe.

Did openqa_clone_job_o3 --skip-chained-deps 1503691 INCLUDE_MODULES=isosize ISO_MAXSIZE=42 TEST=okurz_poo80830 _GROUP=0 BUILD= _SKIP_POST_FAIL_HOOKS=1 which triggered https://openqa.opensuse.org/tests/1509703# and the minion job https://openqa.opensuse.org/minion/jobs?id=350616 which shows in "hook_result" of the minion job output details: Job w/o job group, \$exclude_no_group is set, skipping investigation\n. Not beautiful but working :)

Next test case with "development job group" which should be excluded as well:

openqa_clone_job_o3 --skip-chained-deps 1503691 INCLUDE_MODULES=isosize ISO_MAXSIZE=42 TEST=okurz_poo80830 _GROUP_ID=38 BUILD= _SKIP_POST_FAIL_HOOKS=1

Created job #1509704: opensuse-Tumbleweed-NET-aarch64-Build20201211-textmode@aarch64 -> https://openqa.opensuse.org/t1509704
where group id "38" is Development / Development Tumbleweed.

This yields https://openqa.opensuse.org/minion/jobs?id=350617 with "Job group 'Development Tumbleweed / Development' matches \$exclude_group_regex '(Development|Open Build Service|Others)./.(Kernel|Development|Staging|MicroOS)', skipping investigation\n". Also nice, as expected.

Next test in a valid job group that is not ignored:

openqa_clone_job_o3 --skip-chained-deps 1503691 INCLUDE_MODULES=isosize ISO_MAXSIZE=42 TEST=okurz_poo80830 _GROUP_ID=24 BUILD= _SKIP_POST_FAIL_HOOKS=1

Created job #1509705: opensuse-Tumbleweed-NET-aarch64-Build20201211-textmode@aarch64 -> https://openqa.opensuse.org/t1509705
where group id "24" is openQA with the minion details https://openqa.opensuse.org/minion/jobs?id=350618 "hook_result" => "{\n \"id\" : 101174\n}\n" which is the comment id within https://openqa.opensuse.org/tests/1509705#comment-101174 with

Automatic investigation jobs:
    okurz_poo80830:investigate:retry: No last good recorded, skipping regression investigation jobs

also nice. I have deleted the latest job again for cleanup so that it does not pollute the job group.

I am not yet fully convinced this works all stable so I want to monitor the situation further.

TODO:

• Monitor after some days if jobs are also triggered for example in Tumbleweed aarch64, e.g. https://openqa.opensuse.org/tests/latest?arch=aarch64&distri=opensuse&flavor=DVD&machine=aarch64&test=desktopapps-documentation-x11&version=Tumbleweed
• With the SQL query on o3 select jobs.id, result_dir,t_finished from comments,jobs,users where comments.user_id = users.id and comments.job_id = jobs.id and nickname ~ 'geekotest' and text ~ 'Automatic investigation jobs' order by id DESC limit 10; check if new comments are posted

Bumping due date to account for holidays

Actually with the PR merged everything is done :)