QA » openQA Project » openQA Infrastructure

On openqaworker1 from /var/log/zypp/history I can find that some packages changed:

2019-06-19 00:22:10|install|os-autoinst|4.5.1560860907.6a126c01-lp150.122.1|x86_64||devel_openQA|80914acf82e97567ebaf15155edc7f91d3f75ef041eab7ce73e2176e449ba2b4|
2019-06-19 00:22:13|install|os-autoinst-openvswitch|4.5.1560860907.6a126c01-lp150.122.1|x86_64||devel_openQA|6dfdcb409a9bf9e1fdd90c6f958ee806cc233203326d044481d3270bdb2c1244|
2019-06-19 00:22:14|install|openQA-common|4.6.1560873517.08f60a7b-lp150.1454.1|noarch||devel_openQA|eebebcc7eca4cb8f06150fd0a40b99d863be1f609a86b4f7675d41644e12301b|
2019-06-19 00:22:15|install|openQA-client|4.6.1560873517.08f60a7b-lp150.1454.1|noarch||devel_openQA|6a51abd2e667b8f3d20dae00105d2a4f4bcb52dc8cdfc448347f88046bf375c5|
2019-06-19 00:22:17|install|openQA-worker|4.6.1560873517.08f60a7b-lp150.1454.1|noarch||devel_openQA|0ad44d082e2bdbd07bcf6cdd13b7e1ede17ccb55be9321003790bec90e59de76|
2019-06-19 00:23:14|install|kernel-default|4.12.14-lp150.12.64.1|x86_64||repo-update|18db73d2895d43bf08e3d7ccfb62bce4e754055ee5b5c9b6db6c909d29baa873|

forcing the installation of "perl-DBIx-Class-DeploymentHandler" fixes it. I tried that out manually on power8.o.o and ensured that workers can restart.

The whole change within the openQA package from yesterday to today seems to be:

$ git log --no-merges 7c663c0b..08f60a7b
commit fe417329
Author: Sebastian Riedel <sri@cpan.org>
Date:   Tue Jun 18 16:48:05 2019 +0200

    Calculate elapsed time more correctly

commit 7e56fd2c
Author: Sebastian Riedel <sri@cpan.org>
Date:   Tue Jun 18 15:51:20 2019 +0200

    Add some documentation for OpenQA::Schema::Profiler

commit b05487dd
Author: Sebastian Riedel <sri@cpan.org>
Date:   Tue Jun 18 15:39:57 2019 +0200

    Refactor DBIx::Class profiler to use current openQA best practices

commit 2f06936b
Author: Sebastian Riedel <sri@cpan.org>
Date:   Mon Jun 17 15:00:34 2019 +0200

    Remove dead code and move rng functions still in use to OpenQA::Utils

commit dc2dacb7 (okurz/enhance/multi_machine_doc, enhance/multi_machine_doc)
Author: Oliver Kurz <okurz@suse.de>
Date:   Tue Jun 18 06:23:25 2019 +0200

    docs: Fix shell code box rendering with line break

I am suspecting https://github.com/os-autoinst/openQA/pull/2118

But now the transactional-workers fail with

Jun 19 07:01:55 openqaworker1 worker[3536]: {UNKNOWN}: Permission denied
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Assets.pm line 28.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Assets.pm line 28.
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/lib/perl5/vendor_perl/5.26.1/Class/C3/Componentised.pm line 146. at /usr/lib/perl5/vendor_perl/5.26.1/Class/C3/Componentised.pm line 151
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Schema/Profiler.pm line 23.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Schema/Profiler.pm line 23.
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Setup.pm line 25.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Setup.pm line 25.
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Worker.pm line 34.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Worker.pm line 34.
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/worker line 106.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/worker line 106.

Looks like something wants to rebuild code within the read-only root dir on the transactional-workers.
Looks like again something wants to access stuff which is not allowed by apparmor:

type=AVC msg=audit(1560921323.698:5407): apparmor="ALLOWED" operation="open" profile="/usr/share/openqa/script/worker" name="/usr/bin/unzip-plain" pid=13345 comm="worker" requested_mask="r" denied_mask="r" fsuid=471 ouid=0
…
type=AVC msg=audit(1560921325.638:5419): apparmor="ALLOWED" operation="open" profile="/usr/share/openqa/script/worker" name="/usr/bin/unzip-plain" pid=13377 comm="worker" requested_mask="r" denied_mask="r" fsuid=471 ouid=0

Temporarily fixed on the workers with

for i in power8 aarch64 imagetester openqaworker1 openqaworker4; do echo $i && ssh root@$i "aa-complain /usr/share/openqa/script/worker && systemctl restart openqa-worker.target"; done

and created https://github.com/os-autoinst/openQA/pull/2119 for two (potential) fixes.

Think i know how to fix it. https://github.com/os-autoinst/openQA/pull/2120

https://github.com/os-autoinst/openQA/pull/2119 merged however we agreed that he database migration should never be triggered by the worker hence https://github.com/os-autoinst/openQA/pull/2120 , already merged.

Also, we can investigate if we can not move the dependency on DBIx::Class to the worker in the .spec file.

Trying to have a simple "worker smoke test" based on local packages with

docker run -it --rm -v /home/okurz/local/tmp:/tmp/local/tmp:ro opensuse/tumbleweed sh -c 'zypper --no-refresh in --allow-unsigned-rpm /tmp/local/tmp/openQA-{client,common,worker}-4.6.1559830256.1da81757-0.noarch.rpm && /usr/share/openqa/script/worker --verbose --instance 1'

and in the .spec file I moved the DBIx dependencies out of %t_requires into the section for the main package, being webUI.

https://github.com/os-autoinst/openQA/pull/2124 should be the final PR in this ticket removing the dependency on DBIx::Class in the worker package completely.

What I do not understand is why my package "openQA-common" still depends on DBIx::Class:

$ for i in ~/local/tmp/openQA*-4.6.1560937225.eb276beb-0.noarch.rpm ; do echo "### $i ###" && (rpm -qpR $i | grep DBI) ; done
### /home/okurz/local/tmp/openQA-4.6.1560937225.eb276beb-0.noarch.rpm ###
perl(DBIx::Class) >= 0.082801
perl(DBIx::Class::DeploymentHandler)
### /home/okurz/local/tmp/openQA-bootstrap-4.6.1560937225.eb276beb-0.noarch.rpm ###
### /home/okurz/local/tmp/openQA-client-4.6.1560937225.eb276beb-0.noarch.rpm ###
### /home/okurz/local/tmp/openQA-common-4.6.1560937225.eb276beb-0.noarch.rpm ###
perl(DBIx::Class)
perl(DBIx::Class::DynamicDefault)
perl(DBIx::Class::OptimisticLocking)
perl(DBIx::Class::Schema::Config)
perl(DBIx::Class::Storage::Statistics)
### /home/okurz/local/tmp/openQA-doc-4.6.1560937225.eb276beb-0.noarch.rpm ###
### /home/okurz/local/tmp/openQA-local-db-4.6.1560937225.eb276beb-0.noarch.rpm ###
### /home/okurz/local/tmp/openQA-worker-4.6.1560937225.eb276beb-0.noarch.rpm ###