action #53285
closedall o3 workers fail with "Can't locate DBIx/Class/DeploymentHandler.pm"
0%
Updated by okurz almost 5 years ago
On openqaworker1 from /var/log/zypp/history I can find that some packages changed:
2019-06-19 00:22:10|install|os-autoinst|4.5.1560860907.6a126c01-lp150.122.1|x86_64||devel_openQA|80914acf82e97567ebaf15155edc7f91d3f75ef041eab7ce73e2176e449ba2b4|
2019-06-19 00:22:13|install|os-autoinst-openvswitch|4.5.1560860907.6a126c01-lp150.122.1|x86_64||devel_openQA|6dfdcb409a9bf9e1fdd90c6f958ee806cc233203326d044481d3270bdb2c1244|
2019-06-19 00:22:14|install|openQA-common|4.6.1560873517.08f60a7b-lp150.1454.1|noarch||devel_openQA|eebebcc7eca4cb8f06150fd0a40b99d863be1f609a86b4f7675d41644e12301b|
2019-06-19 00:22:15|install|openQA-client|4.6.1560873517.08f60a7b-lp150.1454.1|noarch||devel_openQA|6a51abd2e667b8f3d20dae00105d2a4f4bcb52dc8cdfc448347f88046bf375c5|
2019-06-19 00:22:17|install|openQA-worker|4.6.1560873517.08f60a7b-lp150.1454.1|noarch||devel_openQA|0ad44d082e2bdbd07bcf6cdd13b7e1ede17ccb55be9321003790bec90e59de76|
2019-06-19 00:23:14|install|kernel-default|4.12.14-lp150.12.64.1|x86_64||repo-update|18db73d2895d43bf08e3d7ccfb62bce4e754055ee5b5c9b6db6c909d29baa873|
forcing the installation of "perl-DBIx-Class-DeploymentHandler" fixes it. I tried that out manually on power8.o.o and ensured that workers can restart.
The whole change within the openQA package from yesterday to today seems to be:
$ git log --no-merges 7c663c0b..08f60a7b
commit fe417329
Author: Sebastian Riedel <sri@cpan.org>
Date: Tue Jun 18 16:48:05 2019 +0200
Calculate elapsed time more correctly
commit 7e56fd2c
Author: Sebastian Riedel <sri@cpan.org>
Date: Tue Jun 18 15:51:20 2019 +0200
Add some documentation for OpenQA::Schema::Profiler
commit b05487dd
Author: Sebastian Riedel <sri@cpan.org>
Date: Tue Jun 18 15:39:57 2019 +0200
Refactor DBIx::Class profiler to use current openQA best practices
commit 2f06936b
Author: Sebastian Riedel <sri@cpan.org>
Date: Mon Jun 17 15:00:34 2019 +0200
Remove dead code and move rng functions still in use to OpenQA::Utils
commit dc2dacb7 (okurz/enhance/multi_machine_doc, enhance/multi_machine_doc)
Author: Oliver Kurz <okurz@suse.de>
Date: Tue Jun 18 06:23:25 2019 +0200
docs: Fix shell code box rendering with line break
I am suspecting https://github.com/os-autoinst/openQA/pull/2118
But now the transactional-workers fail with
Jun 19 07:01:55 openqaworker1 worker[3536]: {UNKNOWN}: Permission denied
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Assets.pm line 28.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Assets.pm line 28.
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/lib/perl5/vendor_perl/5.26.1/Class/C3/Componentised.pm line 146. at /usr/lib/perl5/vendor_perl/5.26.1/Class/C3/Componentised.pm line 151
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Schema/Profiler.pm line 23.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Schema/Profiler.pm line 23.
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Setup.pm line 25.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Setup.pm line 25.
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/../lib/OpenQA/Worker.pm line 34.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/../lib/OpenQA/Worker.pm line 34.
Jun 19 07:01:55 openqaworker1 worker[3536]: Compilation failed in require at /usr/share/openqa/script/worker line 106.
Jun 19 07:01:55 openqaworker1 worker[3536]: BEGIN failed--compilation aborted at /usr/share/openqa/script/worker line 106.
Looks like something wants to rebuild code within the read-only root dir on the transactional-workers.
Looks like again something wants to access stuff which is not allowed by apparmor:
type=AVC msg=audit(1560921323.698:5407): apparmor="ALLOWED" operation="open" profile="/usr/share/openqa/script/worker" name="/usr/bin/unzip-plain" pid=13345 comm="worker" requested_mask="r" denied_mask="r" fsuid=471 ouid=0
…
type=AVC msg=audit(1560921325.638:5419): apparmor="ALLOWED" operation="open" profile="/usr/share/openqa/script/worker" name="/usr/bin/unzip-plain" pid=13377 comm="worker" requested_mask="r" denied_mask="r" fsuid=471 ouid=0
Temporarily fixed on the workers with
for i in power8 aarch64 imagetester openqaworker1 openqaworker4; do echo $i && ssh root@$i "aa-complain /usr/share/openqa/script/worker && systemctl restart openqa-worker.target"; done
and created https://github.com/os-autoinst/openQA/pull/2119 for two (potential) fixes.
Updated by kraih almost 5 years ago
Think i know how to fix it. https://github.com/os-autoinst/openQA/pull/2120
Updated by okurz almost 5 years ago
https://github.com/os-autoinst/openQA/pull/2119 merged however we agreed that he database migration should never be triggered by the worker hence https://github.com/os-autoinst/openQA/pull/2120 , already merged.
Also, we can investigate if we can not move the dependency on DBIx::Class to the worker in the .spec file.
Trying to have a simple "worker smoke test" based on local packages with
docker run -it --rm -v /home/okurz/local/tmp:/tmp/local/tmp:ro opensuse/tumbleweed sh -c 'zypper --no-refresh in --allow-unsigned-rpm /tmp/local/tmp/openQA-{client,common,worker}-4.6.1559830256.1da81757-0.noarch.rpm && /usr/share/openqa/script/worker --verbose --instance 1'
and in the .spec file I moved the DBIx dependencies out of %t_requires into the section for the main package, being webUI.
Updated by okurz almost 5 years ago
https://github.com/os-autoinst/openQA/pull/2124 should be the final PR in this ticket removing the dependency on DBIx::Class in the worker package completely.
What I do not understand is why my package "openQA-common" still depends on DBIx::Class:
$ for i in ~/local/tmp/openQA*-4.6.1560937225.eb276beb-0.noarch.rpm ; do echo "### $i ###" && (rpm -qpR $i | grep DBI) ; done
### /home/okurz/local/tmp/openQA-4.6.1560937225.eb276beb-0.noarch.rpm ###
perl(DBIx::Class) >= 0.082801
perl(DBIx::Class::DeploymentHandler)
### /home/okurz/local/tmp/openQA-bootstrap-4.6.1560937225.eb276beb-0.noarch.rpm ###
### /home/okurz/local/tmp/openQA-client-4.6.1560937225.eb276beb-0.noarch.rpm ###
### /home/okurz/local/tmp/openQA-common-4.6.1560937225.eb276beb-0.noarch.rpm ###
perl(DBIx::Class)
perl(DBIx::Class::DynamicDefault)
perl(DBIx::Class::OptimisticLocking)
perl(DBIx::Class::Schema::Config)
perl(DBIx::Class::Storage::Statistics)
### /home/okurz/local/tmp/openQA-doc-4.6.1560937225.eb276beb-0.noarch.rpm ###
### /home/okurz/local/tmp/openQA-local-db-4.6.1560937225.eb276beb-0.noarch.rpm ###
### /home/okurz/local/tmp/openQA-worker-4.6.1560937225.eb276beb-0.noarch.rpm ###
Updated by okurz almost 5 years ago
- Status changed from Feedback to Resolved
Enabled apparmor completely on all workers again. Tests are fine. The current packages are also fine. https://build.opensuse.org/request/show/711804 is the last SR accepted into Factory that has the fixes included. The last mentioned PR is still open: https://github.com/os-autoinst/openQA/pull/2124 As we can also decide we want to keep the additional requirements in the worker package and close the PR we can also close the ticket without further waiting.