action #95179
closed[sporadic] containers: eventually the tests fails on single_container_webui step with the error "Mojolicious::Plugin::AssetPack::Pipe::Sass requires"
Description
Motivation¶
Eventually we saw this error in the container tests.
Can not create MOJO_TMPDIR : mkdir /var/lib/openqa/share/factory/tmp: Permission denied at /usr/share/openqa/script/../lib/OpenQA/Setup.pm line 316.
- https://openqa.opensuse.org/tests/1830297#step/single_container_webui/107
- https://openqa.opensuse.org/tests/1778677#step/single_container_webui/107
Acceptance Criteria¶
- AC 1: The test is fixed and doesn't break here
Steps to reproduce¶
- Find openqa tests failing on https://openqa.opensuse.org/tests/1830297#next_previous
Suggestions¶
- Try to reproduce this can locally
- Try to fix it in openQA itself not just a package
- The same menssage is logged by two different components, maybe is a concurrency problem
Updated by ilausuch over 3 years ago
- Subject changed from cotainers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" to containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp"
Updated by ilausuch over 3 years ago
- Subject changed from containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" to [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by ilausuch over 3 years ago
- Status changed from Workable to In Progress
- Assignee set to ilausuch
Updated by ilausuch over 3 years ago
Working on the investigation, I found something that could be only circumstantial but maybe not.
The run_openqa.sh before we adapt that to have two versions (one for all services together and other to have this split and ready for HA) didn’t execute livehandle (see https://github.com/os-autoinst/openQA/commit/e4e4d4973b2dcc2e02ae136da91d688ea6417c8d#diff-601001db02292ee9c9a3c7f4dbf8699b9465ee448277cef81ecebd2bb83b7a58)
start_daemon -u geekotest /usr/share/openqa/script/openqa-scheduler &
start_daemon -u geekotest /usr/share/openqa/script/openqa-websockets &
start_daemon -u geekotest /usr/share/openqa/script/openqa gru -m production run &
apache2ctl start
start_daemon -u geekotest /usr/share/openqa/script/openqa prefork -m production –proxy
And now this runs the livehandler also
start_daemon -u geekotest /usr/share/openqa/script/openqa-scheduler &
start_daemon -u geekotest /usr/share/openqa/script/openqa-websockets &
start_daemon -u geekotest /usr/share/openqa/script/openqa-livehandler &
start_daemon -u geekotest /usr/share/openqa/script/openqa gru -m production run &
apache2ctl start
start_daemon -u geekotest /usr/share/openqa/script/openqa prefork -m production --proxy
At the same time we can see in the errors of https://openqa.opensuse.org/tests/1837881#step/single_container_webui/108
# Test died: output not validating at openqa/lib/utils.pm line 101.
This line (https://github.com/os-autoinst/openQA/blob/1248f9f088268759cbed5c91104c7ddb29d8c9d8/lib/OpenQA/Utils.pm#L101) refers to a this piece of code
# returns the url to the web socket proxy started via openqa-livehandler
sub determine_web_ui_web_socket_url {
my ($job_id) = @_;
return "liveviewhandler/tests/$job_id/developer/ws-proxy";
}
So this could be a coincidence or not
Updated by openqa_review about 3 years ago
- Due date set to 2021-07-28
Setting due date based on mean cycle time of SUSE QE Tools
Updated by mkittler about 3 years ago
The directory is only created by the web UI but not other services like the livehandler. So I don't think the introduction of livehandler makes a difference.
More importantly, I'd like to note that "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp"
is not the reason why the test fails. It is just a log message which even appears on successful runs: https://openqa.opensuse.org/tests/1839850#step/single_container_webui/8
(The same counts for AH00558: httpd-prefork: Could not reliably determine the server's fully qualified domain name, using 172.18.0.3. Set the 'ServerName' directive globally to suppress this message
.)
The last log message is Mojolicious::Plugin::AssetPack::Pipe::Sass requires ruby. https://ruby-lang.org/en/documentation/installation
which is also absent in successful runs. So maybe that's the culprit? Of course it leaves the question why ruby is attempted to be used in some cases. I would assume it is only a build/development requirement as the asset cache is supplied by the rpm package.
Updated by ilausuch about 3 years ago
For now I created this PR to ensure that the permissions are good for geekotest user
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/71
I tested locally that I the permissions were incorrect and was imposible to create these directories. But I don't know how in openqa only eventually this failed. So independent that is the real problem or not here, I think this solution could help if in the future we test more deeply
Now I am going to investigate your suggestions
Updated by ilausuch about 3 years ago
About the message
Mojolicious::Plugin::AssetPack::Pipe::Sass requires ruby. https://ruby-lang.org/en/documentation/installation
The url is wrong, should be
https://www.ruby-lang.org/en/documentation/installation/
Updated by ilausuch about 3 years ago
https://github.com/os-autoinst/openQA/pull/4050
I am preparing a test as a script in tools to try to reproduce this locally and to have a sooner than in openqa and before merge
Updated by ilausuch about 3 years ago
Now the PR is merged. Waiting for some completed tests in https://openqa.opensuse.org/group_overview/24
Updated by ilausuch about 3 years ago
- Subject changed from [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" size:M to [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Mojolicious::Plugin::AssetPack::Pipe::Sass requires"
- Status changed from In Progress to Resolved
I changed the original title of this ticket to reflect the real reason of the failure.
And this ticket now is resolved and checked that since the PR was merged (we had more than 24 hours of testing without failures with this error) see: https://openqa.opensuse.org/group_overview/24
However the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" still appears on the logs but it is covered by the ticket #95768