action #124739
closedo3 deployment failed due to "run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27" and then "run(/bin/sass) failed: No such file or directory" size:M
Description
Observation¶
Dominique Leuenberger brought up in https://suse.slack.com/archives/C02CANHLANP/p1676644670756539
(Dominique Leuenberger) openQA going down on a friday afternoon/evening? Hope that was not a deployment gone wrong
(Dominique Leuenberger sadly - that seems exactly what's happening: 2023-02-17 14:33:21|command|root@ariel|'zypper' '-n' '--no-refresh' 'dup' '--auto-agree-with-licenses' '--download-in-advance'|
(Oliver Kurz) let me check that. openqa-webui-daemon[16374]: run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27.
(Oliver Kurz) https://openqa.opensuse.org/ up again. Will report a ticket and handle in there.
(Oliver Kurz) by the way we have continuous deployment of openQA+os-autoinst so this was triggered automatically. I guess it was an acceptance of some AssetPacke related package that triggered a build chain and eventually the deployment
(Sebastian Riedel) Sass dependency issue?
(Oliver Kurz) two issues actually, apparmor preventing to run sass and sass was not installed.
I logged into o3 and found the openqa-webui service failed with the error as above about permission denied. So I called
aa-complain /etc/apparmor.d/usr.share.openqa.script.openqa && systemctl start openqa-webui; journalctl -f -u openqa-webui
which showed
Feb 17 14:49:59 ariel systemd[1]: Started The openQA web UI.
…
Feb 17 14:50:02 ariel openqa-webui-daemon[26137]: run(/bin/sass) failed: No such file or directory at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27.
then I did
zypper -n in ruby2.5-rubygem-sass && systemctl restart openqa-webui; journalctl -f -u openqa-webui
that worked. So two issues actually, apparmor preventing to run sass and sass was not installed. I switched back to aa-enforce, restarted and again the permission denied problem.
Acceptance criteria¶
- AC1: Original issue has been identified and mitigated for all users, not just o3
Suggestions¶
- Identify what caused the actual issue
- Fix the issue for all users, not just the o3 case -> for now solved with revert https://github.com/os-autoinst/openQA/pull/5011
Add "sass" to apparmor filesEnsure sass is pulled in as dependency where necessaryRunaa-enforce /etc/apparmor.d/usr.share.openqa.script.openqa
again and ensure the webUI to be properly running- Remove sass from o3 again
- Wait for actual fix in #122440 and check inside those OBS checks for the error
- Ensure that the script generate-packed-assets actually fails on error and OBS check should also fail!
- Cleanup not needed temporary OBS repos
Updated by okurz over 1 year ago
- Status changed from New to In Progress
- Assignee set to okurz
Updated by okurz over 1 year ago
https://github.com/os-autoinst/openQA/pull/5010 to address the apparmor problem. I am not sure if this is enough as the apparmor profile difference on o3 vs. the version in the git repo is rather big.
Updated by kraih over 1 year ago
Given the timing, the problem has to be related to https://github.com/os-autoinst/openQA/pull/5009. There were no big changes in the new perl-Mojolicious-Plugin-AssetPack release (just download retry).
Updated by okurz over 1 year ago
kraih wrote:
Given the timing, the problem has to be related to https://github.com/os-autoinst/openQA/pull/5009. There were no big changes in the new perl-Mojolicious-Plugin-AssetPack release (just download retry).
https://github.com/os-autoinst/openQA/pull/5009 couldn't have triggered a deployment at that time. I'd rather suspect https://github.com/os-autoinst/openQA/pull/5003
and from the zypper log we know:
2023-02-17 14:33:30|install|openQA|4.6.1676642742.2bcc2ae-lp154.5587.1|x86_64||devel_openQA|8e7c549793c8718a37f8262602937f7405253e6c739c747adfa48f6329f55af2|
which is the commit 2bcc2ae https://github.com/os-autoinst/openQA/pull/5008 "Retry asset downloads automatically". I don't understand that enough to be able how that could cause a regression but I still suggest to revert https://github.com/os-autoinst/openQA/pull/5008
Updated by mkittler over 1 year ago
Add "sass" to apparmor files
Ensure sass is pulled in as dependency where necessary
If it runs sass, then something is broken. At least when I remember correctly, it should only ever run sass when there's a cache miss but our package should provide a "perfect" cache.
Updated by okurz over 1 year ago
- Priority changed from Immediate to High
We used a container with
podman run --rm -it registry.opensuse.org/opensuse/tumbleweed
and in there
zypper -n in -t pattern apparmor && zypper -n in openQA-local-db sudo
sudo -u postgres /usr/share/postgresql/postgresql-script start
sudo -u postgres /usr/share/openqa/script/setup-db
sudo -u geekotest /usr/share/openqa/script/openqa-webui-daemon
this starts just fine. To reproduce the problem:
zypper ar -p 95 -f 'http://download.opensuse.org/repositories/devel:openQA/openSUSE_Tumbleweed' devel_openQA && zypper dup --allow-vendor-change && sudo -u geekotest /usr/share/openqa/script/openqa-webui-daemon
this shows
Mojolicious::Plugin::AssetPack::Pipe::Sass requires ruby. https://ruby-lang.org/en/documentation/installation
so a nicer message than what we saw on o3 or what mkittler could reproduce locally in his Tumbleweed but we assume actually the same error.
The git log between the last good in Tumbleweed and the first bad in devel:openQA:
$ git log1 --no-merges 8d9ce6f..8a49082 4b6067931 Enable build for PowerPC after openQA packages are no longer noarch
242fd8f74 Retry asset downloads automatically
dde33e0b3 Add pagination for GET /api/v1/workers
3416c917e Remove unused 'backend_info'
7ed949e88 Track coverage of test server spawned by `t/25-downloader.t`
e4a4bd88b Avoid caveats of `Archive::Extract` using `bsdtar` instead
e26aa3822 Remove noarch from openQA.spec
03ce60f98 Dependency cron 2023-02-15
00b0f0b28 Update documentation of bug references and labels to meanwhile changes
284d1e1de Fix investigation git log
8fa363454 Render bugrefs as clickable links within labels
ee74a4c30 Use signatures in Markdown module
f23d89090 Ensure consistent openQA version when installing openQA-local-db
I was suspecting the "noarch" change to trigger a change but why would the problem only happen now?:
Anyway, I did create a PR to revert the "noarch" change.
shows
[ 69s] ./tools/generate-packed-assets
[ 69s] Could not load "Mojolicious::Plugin::AssetPack::Pipe::if": Can't locate Mojolicious/Plugin/AssetPack/Pipe/if.pm in @INC (you may need to install the Mojolicious::Plugin::AssetPack::Pipe::if module) (@INC contains: lib/ /usr/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.36.0 /usr/lib/perl5/vendor_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.36.0 /usr/lib/perl5/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/5.36.0 /usr/lib/perl5/site_perl) at (eval 142) line 1.
[ 69s] at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack/Util.pm line 48.
[ 69s] Mojolicious::Plugin::AssetPack::Util::load_module("Mojolicious::Plugin::AssetPack::Pipe::if") called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 112
[ 69s] Mojolicious::Plugin::AssetPack::_pipes(Mojolicious::Plugin::AssetPack=HASH(0x5609bc974af0), ARRAY(0x5609bc9484b8)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 80
[ 69s] Mojolicious::Plugin::AssetPack::register(Mojolicious::Plugin::AssetPack=HASH(0x5609bc974af0), Mojolicious::Lite=HASH(0x5609bca65810), HASH(0x5609bc974b20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugins.pm line 45
[ 69s] Mojolicious::Plugins::register_plugin(Mojolicious::Plugins=HASH(0x5609bd51b188), "AssetPack", Mojolicious::Lite=HASH(0x5609bca65810), HASH(0x5609bc974b20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious.pm line 175
[ 69s] Mojolicious::plugin(Mojolicious::Lite=HASH(0x5609bca65810), "AssetPack", HASH(0x5609bc974b20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Lite.pm line 45
[ 69s] main::plugin("AssetPack", HASH(0x5609bc974b20)) called at -e line 1
[ 71s] Could not load "Mojolicious::Plugin::AssetPack::Pipe::if": Can't locate Mojolicious/Plugin/AssetPack/Pipe/if.pm in @INC (you may need to install the Mojolicious::Plugin::AssetPack::Pipe::if module) (@INC contains: lib/ /usr/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.36.0 /usr/lib/perl5/vendor_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.36.0 /usr/lib/perl5/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/5.36.0 /usr/lib/perl5/site_perl) at (eval 142) line 1.
[ 71s] at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack/Util.pm line 48.
[ 71s] Mojolicious::Plugin::AssetPack::Util::load_module("Mojolicious::Plugin::AssetPack::Pipe::if") called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 112
[ 71s] Mojolicious::Plugin::AssetPack::_pipes(Mojolicious::Plugin::AssetPack=HASH(0x556f5f34aaf0), ARRAY(0x556f5f31e4b8)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 80
[ 71s] Mojolicious::Plugin::AssetPack::register(Mojolicious::Plugin::AssetPack=HASH(0x556f5f34aaf0), Mojolicious::Lite=HASH(0x556f5f43b810), HASH(0x556f5f34ab20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugins.pm line 45
[ 71s] Mojolicious::Plugins::register_plugin(Mojolicious::Plugins=HASH(0x556f5fef1188), "AssetPack", Mojolicious::Lite=HASH(0x556f5f43b810), HASH(0x556f5f34ab20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious.pm line 175
[ 71s] Mojolicious::plugin(Mojolicious::Lite=HASH(0x556f5f43b810), "AssetPack", HASH(0x556f5f34ab20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Lite.pm line 45
[ 71s] main::plugin("AssetPack", HASH(0x556f5f34ab20)) called at -e line 1
[ 72s] Could not load "Mojolicious::Plugin::AssetPack::Pipe::if": Can't locate Mojolicious/Plugin/AssetPack/Pipe/if.pm in @INC (you may need to install the Mojolicious::Plugin::AssetPack::Pipe::if module) (@INC contains: lib/ /usr/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.36.0 /usr/lib/perl5/vendor_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.36.0 /usr/lib/perl5/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/5.36.0 /usr/lib/perl5/site_perl) at (eval 142) line 1.
[ 72s] at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack/Util.pm line 48.
[ 72s] Mojolicious::Plugin::AssetPack::Util::load_module("Mojolicious::Plugin::AssetPack::Pipe::if") called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 112
[ 72s] Mojolicious::Plugin::AssetPack::_pipes(Mojolicious::Plugin::AssetPack=HASH(0x55c28806faf0), ARRAY(0x55c2880434b8)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 80
[ 72s] Mojolicious::Plugin::AssetPack::register(Mojolicious::Plugin::AssetPack=HASH(0x55c28806faf0), Mojolicious::Lite=HASH(0x55c288160810), HASH(0x55c28806fb20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugins.pm line 45
[ 72s] Mojolicious::Plugins::register_plugin(Mojolicious::Plugins=HASH(0x55c288c16188), "AssetPack", Mojolicious::Lite=HASH(0x55c288160810), HASH(0x55c28806fb20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious.pm line 175
[ 72s] Mojolicious::plugin(Mojolicious::Lite=HASH(0x55c288160810), "AssetPack", HASH(0x55c28806fb20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Lite.pm line 45
[ 72s] main::plugin("AssetPack", HASH(0x55c28806fb20)) called at -e line 1
in comparison to https://build.opensuse.org/build/devel:openQA:tested/openSUSE_Factory/x86_64/openQA/_log which shows
[ 72s] + make install DESTDIR=/home/abuild/rpmbuild/BUILDROOT/openQA-4.6.1676474487.945e502-932.1.x86_64
[ 72s] ./tools/generate-packed-assets
[ 90s] for i in lib public script templates assets; do \
so no error message at that time.
# grep 'install|openQA|' /var/log/zypp/history | tail -n 20
2023-02-08 17:03:36|install|openQA|4.6.1675863678.6b1808c-lp154.5563.2|noarch||devel_openQA|74328c3d9747a9cd7520ab5528d7e8b9e5ee6aaadd17fcce2b11ea985a1c8440|
2023-02-09 03:17:26|install|openQA|4.6.1675910933.095724c-lp154.5564.1|noarch||devel_openQA|300245598ba7d63a393a5bb1d8dc1486d10bb4862ce8c445af8116453787cfcb|
2023-02-09 12:49:36|install|openQA|4.6.1675945253.a4dc44d-lp154.5565.1|noarch||devel_openQA|4b0100b204183955110098de82258cd613c6a8f4bcb7a842eb5f803b58699ad6|
2023-02-10 13:15:32|install|openQA|4.6.1676033243.8d9ce6f-lp154.5566.1|noarch||devel_openQA|ea410380c2390f90c7e4aacac0757a9568c66f4af0c76094831ad09a532114e7|
2023-02-11 02:54:50|install|openQA|4.6.1676033243.8d9ce6f-lp154.5567.1|noarch||devel_openQA|337845f324a7bed66dd58210172597465e6066427a9ebcdf228a29c41be58da5|
2023-02-13 10:32:43|install|openQA|4.6.1676033243.8d9ce6f-lp154.5568.1|noarch||devel_openQA|8940a4bd1800873931d462425dd7df88bc86a2115354f44817a0f02850f03b47|
2023-02-13 11:03:47|install|openQA|4.6.1676033243.8d9ce6f-lp154.5569.1|noarch||devel_openQA|6593651e2961f4c743f4dff8ff07ae1cf3790ea38ff177b7c3921d31a7fc4c54|
2023-02-13 13:20:45|install|openQA|4.6.1676033243.8d9ce6f-lp154.5570.1|noarch||devel_openQA|e8ac30a955413ed02bb8b933fa3ab9f8ff958310f9bb3f40fbc1ef6497222c3f|
2023-02-14 11:08:35|install|openQA|4.6.1676371026.955a5ee-lp154.5571.1|noarch||devel_openQA|94734d9f76d786064e5dddebcf1136adb2432d028d09f2b7c47853f0673b210a|
2023-02-14 14:52:50|install|openQA|4.6.1676383840.69c09ce-lp154.5574.1|noarch||devel_openQA|d6d9eedcb4c4e84bac82e17b82a668e044d8b30b3e105ac98de730982588b392|
2023-02-15 03:20:27|install|openQA|4.6.1676429157.8bedfc2-lp154.5575.1|noarch||devel_openQA|cb018deb2c7279eefc53825e1d4ba07e3b18dda2bb9d1b6c3f49081526d36867|
2023-02-15 14:23:06|install|openQA|4.6.1676468080.251579a-lp154.5576.1|noarch||devel_openQA|43ab0653233268e860485cbeecf45aa012c53d307eeafd57056696422767dc23|
2023-02-15 15:55:35|install|openQA|4.6.1676474487.945e502-lp154.5577.1|x86_64||devel_openQA|744d7c7107fc6b166a836c41eac110d75acd7318812d483547170cdb21b480ed|
2023-02-16 15:01:12|install|openQA|4.6.1676474487.945e502-lp154.5578.1|x86_64||devel_openQA|f0c65e72feb9a820e303976f07de315cc0d12570fe4451ed3a2df44d85417a46|
2023-02-16 16:22:47|install|openQA|4.6.1676474487.945e502-lp154.5579.1|x86_64||devel_openQA|09621ac26298dbbd499931538e50b90a6aa01b0cf42ecde8686b707bdf312be3|
2023-02-16 17:29:05|install|openQA|4.6.1676566786.ac18cc0-lp154.5580.1|x86_64||devel_openQA|1d5671817c5174fe13b0c08b840a9bee60eebd328bc5ceaeae1399acdb51f447|
2023-02-17 08:55:36|install|openQA|4.6.1676621380.f6075b0-lp154.5584.1|x86_64||devel_openQA|d63ce2d3cbbdb00e8ef338ba1c46ac7d9ba3d326692275b62f19b7b96cd1b85a|
2023-02-17 13:15:58|install|openQA|4.6.1676621380.f6075b0-lp154.5585.1|x86_64||devel_openQA|a013556c6e582cc145d5fa36bb5fb21a1744e473efcfb156d692563f1e510cb3|
2023-02-17 14:33:30|install|openQA|4.6.1676642742.2bcc2ae-lp154.5587.1|x86_64||devel_openQA|8e7c549793c8718a37f8262602937f7405253e6c739c747adfa48f6329f55af2|
2023-02-17 16:20:37|install|openQA|4.6.1676646645.8a49082-lp154.5590.1|x86_64||devel_openQA|f83f452e94a5eb43d9580af92e8a505c3289e97a4b10b1831e2b162b28ec1c2b|
shows that likely the last good is actually 13:15:58|install|openQA|4.6.1676621380.f6075b0-lp154.5585.1|
git log1 --no-merges f6075b0..8a49082
$ git log1 --no-merges f6075b0..8a49082
4b6067931 Enable build for PowerPC after openQA packages are no longer noarch
242fd8f74 Retry asset downloads automatically
So created https://github.com/os-autoinst/openQA/pull/5013
and looking into https://build.opensuse.org/package/live_build_log/devel:openQA:GitHub:os-autoinst:openQA:PR-5013/openQA/openSUSE_Tumbleweed/x86_64
we find the same error message:
[ 69s] ./tools/generate-packed-assets
[ 70s] Could not load "Mojolicious::Plugin::AssetPack::Pipe::if": Can't locate Mojolicious/Plugin/AssetPack/Pipe/if.pm in @INC (you may need to install the Mojolicious::Plugin::AssetPack::Pipe::if
so same problem. But in
https://github.com/os-autoinst/openQA/pull/5011
in the OBS check https://build.opensuse.org/package/live_build_log/devel:openQA:GitHub:os-autoinst:openQA:PR-5011/openQA/openSUSE_Tumbleweed/x86_64 we don't see the error message. So actually https://github.com/os-autoinst/openQA/pull/5008 is the culprit even though that obviously does not make any sense :D
Well, phew, we merged that revert. That should mitigate the problem for users using the package.
Updated by okurz over 1 year ago
- Related to action #122440: [sporadic] openQA Assetpack download can fail on initial download size:M added
Updated by tinita over 1 year ago
First aid PR: https://github.com/os-autoinst/openQA/pull/5014
Updated by tinita over 1 year ago
- Copied to action #124757: Move AssetPack plugins into a config file added
Updated by okurz over 1 year ago
- Status changed from In Progress to New
- Assignee deleted (
okurz)
I removed "sass" and dependencies from o3 again, removed the sass stuff from the apparmor profile on o3 and enforced it again. Looks good so far. Please take care about that when bringing in changes again. Unassigning again.
Updated by livdywan over 1 year ago
- Subject changed from o3 deployment failed due to "run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27" and then "run(/bin/sass) failed: No such file or directory" to o3 deployment failed due to "run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27" and then "run(/bin/sass) failed: No such file or directory" size:M
- Description updated (diff)
- Status changed from New to In Progress
- Assignee set to tinita
Updated by tinita over 1 year ago
- Status changed from In Progress to Feedback
Updated by tinita over 1 year ago
- Status changed from Feedback to Resolved
openQA 4.6.1676888404.afe11c5-lp154.5596.1 deployed on o3 and everything seems to be ok