Project

General

Profile

action #124739

Updated by okurz about 1 year ago

## Observation 
 Dominique Leuenberger brought up in https://suse.slack.com/archives/C02CANHLANP/p1676644670756539 
 > (Dominique Leuenberger) openQA going down on a friday afternoon/evening? Hope that was not a deployment gone wrong 
 > (Dominique Leuenberger sadly - that seems exactly what's happening: 2023-02-17 14:33:21|command|root@ariel|'zypper' '-n' '--no-refresh' 'dup' '--auto-agree-with-licenses' '--download-in-advance'| 
 > (Oliver Kurz) let me check that. openqa-webui-daemon[16374]: run(/usr/bin/ruby) failed: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27. 
 > (Oliver Kurz) https://openqa.opensuse.org/ up again. Will report a ticket and handle in there. 
 > (Oliver Kurz) by the way we have continuous deployment of openQA+os-autoinst so this was triggered automatically. I guess it was an acceptance of some AssetPacke related package that triggered a build chain and eventually the deployment 
 > (Sebastian Riedel) Sass dependency issue? 
 > (Oliver Kurz) two issues actually, apparmor preventing to run sass and sass was not installed. 

 I logged into o3 and found the openqa-webui service failed with the error as above about permission denied. So I called 

 ``` 
 aa-complain /etc/apparmor.d/usr.share.openqa.script.openqa && systemctl start openqa-webui; journalctl -f -u openqa-webui 
 ``` 

 which showed 

 ``` 
 Feb 17 14:49:59 ariel systemd[1]: Started The openQA web UI. 
 … 
 Feb 17 14:50:02 ariel openqa-webui-daemon[26137]: run(/bin/sass) failed: No such file or directory at /usr/lib/perl5/vendor_perl/5.26.1/Mojolicious/Plugin/AssetPack/Pipe.pm line 27. 
 ``` 

 then I did 

 ``` 
 zypper -n in ruby2.5-rubygem-sass && systemctl restart openqa-webui; journalctl -f -u openqa-webui 
 ``` 

 that worked. So two issues actually, apparmor preventing to run sass and sass was not installed. I switched back to aa-enforce, restarted and again the permission denied problem. 

 ## Acceptance criteria 
 * **AC1:** Original issue has been identified and mitigated for all users, not just o3 
 * **AC2:** o3 runs the current openQA webUI version with apparmor enforced 

 ## Suggestions 
 * Identify what caused the actual issue 
 * Fix the issue for all users, not just the o3 case -> for now solved with revert https://github.com/os-autoinst/openQA/pull/5011 
 * ~~Add Add "sass" to apparmor files~~ files 
 * ~~Ensure Ensure sass is pulled in as dependency where necessary~~ necessary 
 * Run `aa-enforce /etc/apparmor.d/usr.share.openqa.script.openqa` again and ensure the webUI to be properly running 
 * Remove sass from o3 again 
 * Wait for actual fix in #122440 and check inside those OBS checks for the error 
 * Ensure that the script generate-packed-assets actually fails on error and OBS check should also fail! 
 * Cleanup not needed temporary OBS repos

Back