Project

General

Profile

Actions
Copy link

action #124739

closed

o3 deployment failed due to "run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27" and then "run(/bin/sass) failed: No such file or directory" size:M

Added by okurz almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
tinita
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2023-02-17
Due date:
% Done:

0%

Estimated time:

Description

Observation

Dominique Leuenberger brought up in https://suse.slack.com/archives/C02CANHLANP/p1676644670756539

(Dominique Leuenberger) openQA going down on a friday afternoon/evening? Hope that was not a deployment gone wrong
(Dominique Leuenberger sadly - that seems exactly what's happening: 2023-02-17 14:33:21|command|root@ariel|'zypper' '-n' '--no-refresh' 'dup' '--auto-agree-with-licenses' '--download-in-advance'|
(Oliver Kurz) let me check that. openqa-webui-daemon[16374]: run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27.
(Oliver Kurz) https://openqa.opensuse.org/ up again. Will report a ticket and handle in there.
(Oliver Kurz) by the way we have continuous deployment of openQA+os-autoinst so this was triggered automatically. I guess it was an acceptance of some AssetPacke related package that triggered a build chain and eventually the deployment
(Sebastian Riedel) Sass dependency issue?
(Oliver Kurz) two issues actually, apparmor preventing to run sass and sass was not installed.

I logged into o3 and found the openqa-webui service failed with the error as above about permission denied. So I called

aa-complain /etc/apparmor.d/usr.share.openqa.script.openqa && systemctl start openqa-webui; journalctl -f -u openqa-webui

which showed

Feb 17 14:49:59 ariel systemd[1]: Started The openQA web UI.
…
Feb 17 14:50:02 ariel openqa-webui-daemon[26137]: run(/bin/sass) failed: No such file or directory at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27.

then I did

zypper -n in ruby2.5-rubygem-sass && systemctl restart openqa-webui; journalctl -f -u openqa-webui

that worked. So two issues actually, apparmor preventing to run sass and sass was not installed. I switched back to aa-enforce, restarted and again the permission denied problem.

Acceptance criteria

  • AC1: Original issue has been identified and mitigated for all users, not just o3

Suggestions

  • Identify what caused the actual issue
  • Fix the issue for all users, not just the o3 case -> for now solved with revert https://github.com/os-autoinst/openQA/pull/5011
  • Add "sass" to apparmor files
  • Ensure sass is pulled in as dependency where necessary
  • Run aa-enforce /etc/apparmor.d/usr.share.openqa.script.openqa again and ensure the webUI to be properly running
  • Remove sass from o3 again
  • Wait for actual fix in #122440 and check inside those OBS checks for the error
  • Ensure that the script generate-packed-assets actually fails on error and OBS check should also fail!
  • Cleanup not needed temporary OBS repos

Related issues 2 (0 open — 2 closed)

Related to openQA Project (public) - action #122440: [sporadic] openQA Assetpack download can fail on initial download size:MResolvedkraih2022-12-252023-02-24

Actions
Copied to openQA Project (public) - action #124757: Move AssetPack plugins into a config fileResolvedkraih

Actions
Actions
Copy link

Also available in: Atom PDF