Project

General

Profile

action #124739

o3 deployment failed due to "run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27" and then "run(/bin/sass) failed: No such file or directory" size:M

Added by okurz 4 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
High
Assignee:
tinita
Category:
Concrete Bugs
Target version:
Ready
Start date:
2023-02-17
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

Dominique Leuenberger brought up in https://suse.slack.com/archives/C02CANHLANP/p1676644670756539

(Dominique Leuenberger) openQA going down on a friday afternoon/evening? Hope that was not a deployment gone wrong
(Dominique Leuenberger sadly - that seems exactly what's happening: 2023-02-17 14:33:21|command|root@ariel|'zypper' '-n' '--no-refresh' 'dup' '--auto-agree-with-licenses' '--download-in-advance'|
(Oliver Kurz) let me check that. openqa-webui-daemon[16374]: run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27.
(Oliver Kurz) https://openqa.opensuse.org/ up again. Will report a ticket and handle in there.
(Oliver Kurz) by the way we have continuous deployment of openQA+os-autoinst so this was triggered automatically. I guess it was an acceptance of some AssetPacke related package that triggered a build chain and eventually the deployment
(Sebastian Riedel) Sass dependency issue?
(Oliver Kurz) two issues actually, apparmor preventing to run sass and sass was not installed.

I logged into o3 and found the openqa-webui service failed with the error as above about permission denied. So I called

aa-complain /etc/apparmor.d/usr.share.openqa.script.openqa && systemctl start openqa-webui; journalctl -f -u openqa-webui

which showed

Feb 17 14:49:59 ariel systemd[1]: Started The openQA web UI.
…
Feb 17 14:50:02 ariel openqa-webui-daemon[26137]: run(/bin/sass) failed: No such file or directory at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27.

then I did

zypper -n in ruby2.5-rubygem-sass && systemctl restart openqa-webui; journalctl -f -u openqa-webui

that worked. So two issues actually, apparmor preventing to run sass and sass was not installed. I switched back to aa-enforce, restarted and again the permission denied problem.

Acceptance criteria

  • AC1: Original issue has been identified and mitigated for all users, not just o3

Suggestions

  • Identify what caused the actual issue
  • Fix the issue for all users, not just the o3 case -> for now solved with revert https://github.com/os-autoinst/openQA/pull/5011
  • Add "sass" to apparmor files
  • Ensure sass is pulled in as dependency where necessary
  • Run aa-enforce /etc/apparmor.d/usr.share.openqa.script.openqa again and ensure the webUI to be properly running
  • Remove sass from o3 again
  • Wait for actual fix in #122440 and check inside those OBS checks for the error
  • Ensure that the script generate-packed-assets actually fails on error and OBS check should also fail!
  • Cleanup not needed temporary OBS repos

Related issues

Related to openQA Project - action #122440: [sporadic] openQA Assetpack download can fail on initial download size:MResolved2022-12-252023-02-24

Copied to openQA Project - action #124757: Move AssetPack plugins into a config fileResolved

History

#1 Updated by okurz 4 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz

#2 Updated by okurz 4 months ago

https://github.com/os-autoinst/openQA/pull/5010 to address the apparmor problem. I am not sure if this is enough as the apparmor profile difference on o3 vs. the version in the git repo is rather big.

#3 Updated by kraih 4 months ago

Given the timing, the problem has to be related to https://github.com/os-autoinst/openQA/pull/5009. There were no big changes in the new perl-Mojolicious-Plugin-AssetPack release (just download retry).

#4 Updated by okurz 4 months ago

kraih wrote:

Given the timing, the problem has to be related to https://github.com/os-autoinst/openQA/pull/5009. There were no big changes in the new perl-Mojolicious-Plugin-AssetPack release (just download retry).

https://github.com/os-autoinst/openQA/pull/5009 couldn't have triggered a deployment at that time. I'd rather suspect https://github.com/os-autoinst/openQA/pull/5003

and from the zypper log we know:

2023-02-17 14:33:30|install|openQA|4.6.1676642742.2bcc2ae-lp154.5587.1|x86_64||devel_openQA|8e7c549793c8718a37f8262602937f7405253e6c739c747adfa48f6329f55af2|

which is the commit 2bcc2ae https://github.com/os-autoinst/openQA/pull/5008 "Retry asset downloads automatically". I don't understand that enough to be able how that could cause a regression but I still suggest to revert https://github.com/os-autoinst/openQA/pull/5008

#5 Updated by mkittler 4 months ago

Add "sass" to apparmor files
Ensure sass is pulled in as dependency where necessary

If it runs sass, then something is broken. At least when I remember correctly, it should only ever run sass when there's a cache miss but our package should provide a "perfect" cache.

#6 Updated by okurz 4 months ago

  • Priority changed from Immediate to High

We used a container with

podman run --rm -it registry.opensuse.org/opensuse/tumbleweed

and in there

zypper -n in -t pattern apparmor && zypper -n in openQA-local-db sudo
sudo -u postgres /usr/share/postgresql/postgresql-script start
sudo -u postgres /usr/share/openqa/script/setup-db
sudo -u geekotest /usr/share/openqa/script/openqa-webui-daemon

this starts just fine. To reproduce the problem:

zypper ar -p 95 -f 'http://download.opensuse.org/repositories/devel:openQA/openSUSE_Tumbleweed' devel_openQA && zypper dup --allow-vendor-change && sudo -u geekotest /usr/share/openqa/script/openqa-webui-daemon

this shows

Mojolicious::Plugin::AssetPack::Pipe::Sass requires ruby. https://ruby-lang.org/en/documentation/installation

so a nicer message than what we saw on o3 or what mkittler could reproduce locally in his Tumbleweed but we assume actually the same error.

The git log between the last good in Tumbleweed and the first bad in devel:openQA:

$ git log1 --no-merges 8d9ce6f..8a49082                                                                                                                                                                         4b6067931 Enable build for PowerPC after openQA packages are no longer noarch
242fd8f74 Retry asset downloads automatically
dde33e0b3 Add pagination for GET /api/v1/workers
3416c917e Remove unused 'backend_info'
7ed949e88 Track coverage of test server spawned by `t/25-downloader.t`
e4a4bd88b Avoid caveats of `Archive::Extract` using `bsdtar` instead
e26aa3822 Remove noarch from openQA.spec
03ce60f98 Dependency cron 2023-02-15
00b0f0b28 Update documentation of bug references and labels to meanwhile changes
284d1e1de Fix investigation git log
8fa363454 Render bugrefs as clickable links within labels
ee74a4c30 Use signatures in Markdown module
f23d89090 Ensure consistent openQA version when installing openQA-local-db

I was suspecting the "noarch" change to trigger a change but why would the problem only happen now?:

Anyway, I did create a PR to revert the "noarch" change.

https://build.opensuse.org/package/live_build_log/devel:openQA:GitHub:os-autoinst:openQA:PR-5012/openQA/openSUSE_Tumbleweed/x86_64

shows

[   69s] ./tools/generate-packed-assets
[   69s] Could not load "Mojolicious::Plugin::AssetPack::Pipe::if": Can't locate Mojolicious/Plugin/AssetPack/Pipe/if.pm in @INC (you may need to install the Mojolicious::Plugin::AssetPack::Pipe::if module) (@INC contains: lib/ /usr/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.36.0 /usr/lib/perl5/vendor_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.36.0 /usr/lib/perl5/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/5.36.0 /usr/lib/perl5/site_perl) at (eval 142) line 1.
[   69s]  at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack/Util.pm line 48.
[   69s]    Mojolicious::Plugin::AssetPack::Util::load_module("Mojolicious::Plugin::AssetPack::Pipe::if") called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 112
[   69s]    Mojolicious::Plugin::AssetPack::_pipes(Mojolicious::Plugin::AssetPack=HASH(0x5609bc974af0), ARRAY(0x5609bc9484b8)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 80
[   69s]    Mojolicious::Plugin::AssetPack::register(Mojolicious::Plugin::AssetPack=HASH(0x5609bc974af0), Mojolicious::Lite=HASH(0x5609bca65810), HASH(0x5609bc974b20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugins.pm line 45
[   69s]    Mojolicious::Plugins::register_plugin(Mojolicious::Plugins=HASH(0x5609bd51b188), "AssetPack", Mojolicious::Lite=HASH(0x5609bca65810), HASH(0x5609bc974b20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious.pm line 175
[   69s]    Mojolicious::plugin(Mojolicious::Lite=HASH(0x5609bca65810), "AssetPack", HASH(0x5609bc974b20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Lite.pm line 45
[   69s]    main::plugin("AssetPack", HASH(0x5609bc974b20)) called at -e line 1
[   71s] Could not load "Mojolicious::Plugin::AssetPack::Pipe::if": Can't locate Mojolicious/Plugin/AssetPack/Pipe/if.pm in @INC (you may need to install the Mojolicious::Plugin::AssetPack::Pipe::if module) (@INC contains: lib/ /usr/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.36.0 /usr/lib/perl5/vendor_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.36.0 /usr/lib/perl5/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/5.36.0 /usr/lib/perl5/site_perl) at (eval 142) line 1.
[   71s]  at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack/Util.pm line 48.
[   71s]    Mojolicious::Plugin::AssetPack::Util::load_module("Mojolicious::Plugin::AssetPack::Pipe::if") called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 112
[   71s]    Mojolicious::Plugin::AssetPack::_pipes(Mojolicious::Plugin::AssetPack=HASH(0x556f5f34aaf0), ARRAY(0x556f5f31e4b8)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 80
[   71s]    Mojolicious::Plugin::AssetPack::register(Mojolicious::Plugin::AssetPack=HASH(0x556f5f34aaf0), Mojolicious::Lite=HASH(0x556f5f43b810), HASH(0x556f5f34ab20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugins.pm line 45
[   71s]    Mojolicious::Plugins::register_plugin(Mojolicious::Plugins=HASH(0x556f5fef1188), "AssetPack", Mojolicious::Lite=HASH(0x556f5f43b810), HASH(0x556f5f34ab20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious.pm line 175
[   71s]    Mojolicious::plugin(Mojolicious::Lite=HASH(0x556f5f43b810), "AssetPack", HASH(0x556f5f34ab20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Lite.pm line 45
[   71s]    main::plugin("AssetPack", HASH(0x556f5f34ab20)) called at -e line 1
[   72s] Could not load "Mojolicious::Plugin::AssetPack::Pipe::if": Can't locate Mojolicious/Plugin/AssetPack/Pipe/if.pm in @INC (you may need to install the Mojolicious::Plugin::AssetPack::Pipe::if module) (@INC contains: lib/ /usr/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.36.0 /usr/lib/perl5/vendor_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.36.0 /usr/lib/perl5/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/5.36.0 /usr/lib/perl5/site_perl) at (eval 142) line 1.
[   72s]  at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack/Util.pm line 48.
[   72s]    Mojolicious::Plugin::AssetPack::Util::load_module("Mojolicious::Plugin::AssetPack::Pipe::if") called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 112
[   72s]    Mojolicious::Plugin::AssetPack::_pipes(Mojolicious::Plugin::AssetPack=HASH(0x55c28806faf0), ARRAY(0x55c2880434b8)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugin/AssetPack.pm line 80
[   72s]    Mojolicious::Plugin::AssetPack::register(Mojolicious::Plugin::AssetPack=HASH(0x55c28806faf0), Mojolicious::Lite=HASH(0x55c288160810), HASH(0x55c28806fb20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Plugins.pm line 45
[   72s]    Mojolicious::Plugins::register_plugin(Mojolicious::Plugins=HASH(0x55c288c16188), "AssetPack", Mojolicious::Lite=HASH(0x55c288160810), HASH(0x55c28806fb20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious.pm line 175
[   72s]    Mojolicious::plugin(Mojolicious::Lite=HASH(0x55c288160810), "AssetPack", HASH(0x55c28806fb20)) called at /usr/lib/perl5/vendor_perl/5.36.0/Mojolicious/Lite.pm line 45
[   72s]    main::plugin("AssetPack", HASH(0x55c28806fb20)) called at -e line 1

in comparison to https://build.opensuse.org/build/devel:openQA:tested/openSUSE_Factory/x86_64/openQA/_log which shows

[   72s] + make install DESTDIR=/home/abuild/rpmbuild/BUILDROOT/openQA-4.6.1676474487.945e502-932.1.x86_64
[   72s] ./tools/generate-packed-assets
[   90s] for i in lib public script templates assets; do \

so no error message at that time.

# grep 'install|openQA|' /var/log/zypp/history  | tail -n 20
2023-02-08 17:03:36|install|openQA|4.6.1675863678.6b1808c-lp154.5563.2|noarch||devel_openQA|74328c3d9747a9cd7520ab5528d7e8b9e5ee6aaadd17fcce2b11ea985a1c8440|
2023-02-09 03:17:26|install|openQA|4.6.1675910933.095724c-lp154.5564.1|noarch||devel_openQA|300245598ba7d63a393a5bb1d8dc1486d10bb4862ce8c445af8116453787cfcb|
2023-02-09 12:49:36|install|openQA|4.6.1675945253.a4dc44d-lp154.5565.1|noarch||devel_openQA|4b0100b204183955110098de82258cd613c6a8f4bcb7a842eb5f803b58699ad6|
2023-02-10 13:15:32|install|openQA|4.6.1676033243.8d9ce6f-lp154.5566.1|noarch||devel_openQA|ea410380c2390f90c7e4aacac0757a9568c66f4af0c76094831ad09a532114e7|
2023-02-11 02:54:50|install|openQA|4.6.1676033243.8d9ce6f-lp154.5567.1|noarch||devel_openQA|337845f324a7bed66dd58210172597465e6066427a9ebcdf228a29c41be58da5|
2023-02-13 10:32:43|install|openQA|4.6.1676033243.8d9ce6f-lp154.5568.1|noarch||devel_openQA|8940a4bd1800873931d462425dd7df88bc86a2115354f44817a0f02850f03b47|
2023-02-13 11:03:47|install|openQA|4.6.1676033243.8d9ce6f-lp154.5569.1|noarch||devel_openQA|6593651e2961f4c743f4dff8ff07ae1cf3790ea38ff177b7c3921d31a7fc4c54|
2023-02-13 13:20:45|install|openQA|4.6.1676033243.8d9ce6f-lp154.5570.1|noarch||devel_openQA|e8ac30a955413ed02bb8b933fa3ab9f8ff958310f9bb3f40fbc1ef6497222c3f|
2023-02-14 11:08:35|install|openQA|4.6.1676371026.955a5ee-lp154.5571.1|noarch||devel_openQA|94734d9f76d786064e5dddebcf1136adb2432d028d09f2b7c47853f0673b210a|
2023-02-14 14:52:50|install|openQA|4.6.1676383840.69c09ce-lp154.5574.1|noarch||devel_openQA|d6d9eedcb4c4e84bac82e17b82a668e044d8b30b3e105ac98de730982588b392|
2023-02-15 03:20:27|install|openQA|4.6.1676429157.8bedfc2-lp154.5575.1|noarch||devel_openQA|cb018deb2c7279eefc53825e1d4ba07e3b18dda2bb9d1b6c3f49081526d36867|
2023-02-15 14:23:06|install|openQA|4.6.1676468080.251579a-lp154.5576.1|noarch||devel_openQA|43ab0653233268e860485cbeecf45aa012c53d307eeafd57056696422767dc23|
2023-02-15 15:55:35|install|openQA|4.6.1676474487.945e502-lp154.5577.1|x86_64||devel_openQA|744d7c7107fc6b166a836c41eac110d75acd7318812d483547170cdb21b480ed|
2023-02-16 15:01:12|install|openQA|4.6.1676474487.945e502-lp154.5578.1|x86_64||devel_openQA|f0c65e72feb9a820e303976f07de315cc0d12570fe4451ed3a2df44d85417a46|
2023-02-16 16:22:47|install|openQA|4.6.1676474487.945e502-lp154.5579.1|x86_64||devel_openQA|09621ac26298dbbd499931538e50b90a6aa01b0cf42ecde8686b707bdf312be3|
2023-02-16 17:29:05|install|openQA|4.6.1676566786.ac18cc0-lp154.5580.1|x86_64||devel_openQA|1d5671817c5174fe13b0c08b840a9bee60eebd328bc5ceaeae1399acdb51f447|
2023-02-17 08:55:36|install|openQA|4.6.1676621380.f6075b0-lp154.5584.1|x86_64||devel_openQA|d63ce2d3cbbdb00e8ef338ba1c46ac7d9ba3d326692275b62f19b7b96cd1b85a|
2023-02-17 13:15:58|install|openQA|4.6.1676621380.f6075b0-lp154.5585.1|x86_64||devel_openQA|a013556c6e582cc145d5fa36bb5fb21a1744e473efcfb156d692563f1e510cb3|
2023-02-17 14:33:30|install|openQA|4.6.1676642742.2bcc2ae-lp154.5587.1|x86_64||devel_openQA|8e7c549793c8718a37f8262602937f7405253e6c739c747adfa48f6329f55af2|
2023-02-17 16:20:37|install|openQA|4.6.1676646645.8a49082-lp154.5590.1|x86_64||devel_openQA|f83f452e94a5eb43d9580af92e8a505c3289e97a4b10b1831e2b162b28ec1c2b|

shows that likely the last good is actually 13:15:58|install|openQA|4.6.1676621380.f6075b0-lp154.5585.1|

git log1 --no-merges f6075b0..8a49082

$ git log1 --no-merges f6075b0..8a49082                                                                                                                                                                    
4b6067931 Enable build for PowerPC after openQA packages are no longer noarch
242fd8f74 Retry asset downloads automatically

So created https://github.com/os-autoinst/openQA/pull/5013

and looking into https://build.opensuse.org/package/live_build_log/devel:openQA:GitHub:os-autoinst:openQA:PR-5013/openQA/openSUSE_Tumbleweed/x86_64

we find the same error message:

[   69s] ./tools/generate-packed-assets
[   70s] Could not load "Mojolicious::Plugin::AssetPack::Pipe::if": Can't locate Mojolicious/Plugin/AssetPack/Pipe/if.pm in @INC (you may need to install the Mojolicious::Plugin::AssetPack::Pipe::if

so same problem. But in
https://github.com/os-autoinst/openQA/pull/5011
in the OBS check https://build.opensuse.org/package/live_build_log/devel:openQA:GitHub:os-autoinst:openQA:PR-5011/openQA/openSUSE_Tumbleweed/x86_64 we don't see the error message. So actually https://github.com/os-autoinst/openQA/pull/5008 is the culprit even though that obviously does not make any sense :D

Well, phew, we merged that revert. That should mitigate the problem for users using the package.

#7 Updated by okurz 4 months ago

  • Related to action #122440: [sporadic] openQA Assetpack download can fail on initial download size:M added

#8 Updated by okurz 4 months ago

  • Description updated (diff)

#10 Updated by tinita 4 months ago

  • Copied to action #124757: Move AssetPack plugins into a config file added

#11 Updated by okurz 4 months ago

  • Status changed from In Progress to New
  • Assignee deleted (okurz)

I removed "sass" and dependencies from o3 again, removed the sass stuff from the apparmor profile on o3 and enforced it again. Looks good so far. Please take care about that when bringing in changes again. Unassigning again.

#12 Updated by cdywan 4 months ago

  • Subject changed from o3 deployment failed due to "run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27" and then "run(/bin/sass) failed: No such file or directory" to o3 deployment failed due to "run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27" and then "run(/bin/sass) failed: No such file or directory" size:M
  • Description updated (diff)
  • Status changed from New to In Progress
  • Assignee set to tinita

#13 Updated by tinita 4 months ago

  • Status changed from In Progress to Feedback

#14 Updated by tinita 4 months ago

  • Status changed from Feedback to Resolved

openQA 4.6.1676888404.afe11c5-lp154.5596.1 deployed on o3 and everything seems to be ok

Also available in: Atom PDF