Project

General

Profile

action #95179

[sporadic] containers: eventually the tests fails on single_container_webui step with the error "Mojolicious::Plugin::AssetPack::Pipe::Sass requires"

Added by ilausuch 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2021-07-07
Due date:
2021-07-28
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

Eventually we saw this error in the container tests.

Can not create MOJO_TMPDIR : mkdir /var/lib/openqa/share/factory/tmp: Permission denied at /usr/share/openqa/script/../lib/OpenQA/Setup.pm line 316.

https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=openqa&flavor=dev&machine=64bit-2G&test=openqa_from_containers&version=Tumbleweed

Acceptance Criteria

  • AC 1: The test is fixed and doesn't break here

Steps to reproduce

Suggestions

  • Try to reproduce this can locally
  • Try to fix it in openQA itself not just a package
  • The same menssage is logged by two different components, maybe is a concurrency problem

History

#1 Updated by ilausuch 3 months ago

  • Subject changed from cotainers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" to containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp"

#2 Updated by okurz 3 months ago

  • Target version set to Ready

#3 Updated by ilausuch 2 months ago

  • Subject changed from containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" to [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" size:M
  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by ilausuch 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to ilausuch

#5 Updated by ilausuch 2 months ago

Working on the investigation, I found something that could be only circumstantial but maybe not.

The run_openqa.sh before we adapt that to have two versions (one for all services together and other to have this split and ready for HA) didn’t execute livehandle (see https://github.com/os-autoinst/openQA/commit/e4e4d4973b2dcc2e02ae136da91d688ea6417c8d#diff-601001db02292ee9c9a3c7f4dbf8699b9465ee448277cef81ecebd2bb83b7a58)

start_daemon -u geekotest /usr/share/openqa/script/openqa-scheduler &
start_daemon -u geekotest /usr/share/openqa/script/openqa-websockets &
start_daemon -u geekotest /usr/share/openqa/script/openqa gru -m production run &
apache2ctl start
start_daemon -u geekotest /usr/share/openqa/script/openqa prefork -m production –proxy

And now this runs the livehandler also

start_daemon -u geekotest /usr/share/openqa/script/openqa-scheduler &
start_daemon -u geekotest /usr/share/openqa/script/openqa-websockets &
start_daemon -u geekotest /usr/share/openqa/script/openqa-livehandler &
start_daemon -u geekotest /usr/share/openqa/script/openqa gru -m production run &
apache2ctl start
start_daemon -u geekotest /usr/share/openqa/script/openqa prefork -m production --proxy

At the same time we can see in the errors of https://openqa.opensuse.org/tests/1837881#step/single_container_webui/108

# Test died: output not validating at openqa/lib/utils.pm line 101.

This line (https://github.com/os-autoinst/openQA/blob/1248f9f088268759cbed5c91104c7ddb29d8c9d8/lib/OpenQA/Utils.pm#L101) refers to a this piece of code

# returns the url to the web socket proxy started via openqa-livehandler
sub determine_web_ui_web_socket_url {
my ($job_id) = @_;
return "liveviewhandler/tests/$job_id/developer/ws-proxy";
}

So this could be a coincidence or not

#6 Updated by openqa_review 2 months ago

  • Due date set to 2021-07-28

Setting due date based on mean cycle time of SUSE QE Tools

#7 Updated by mkittler 2 months ago

The directory is only created by the web UI but not other services like the livehandler. So I don't think the introduction of livehandler makes a difference.

More importantly, I'd like to note that "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" is not the reason why the test fails. It is just a log message which even appears on successful runs: https://openqa.opensuse.org/tests/1839850#step/single_container_webui/8
(The same counts for AH00558: httpd-prefork: Could not reliably determine the server's fully qualified domain name, using 172.18.0.3. Set the 'ServerName' directive globally to suppress this message.)

The last log message is Mojolicious::Plugin::AssetPack::Pipe::Sass requires ruby. https://ruby-lang.org/en/documentation/installation which is also absent in successful runs. So maybe that's the culprit? Of course it leaves the question why ruby is attempted to be used in some cases. I would assume it is only a build/development requirement as the asset cache is supplied by the rpm package.

#8 Updated by ilausuch 2 months ago

For now I created this PR to ensure that the permissions are good for geekotest user
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/71

I tested locally that I the permissions were incorrect and was imposible to create these directories. But I don't know how in openqa only eventually this failed. So independent that is the real problem or not here, I think this solution could help if in the future we test more deeply

Now I am going to investigate your suggestions

#9 Updated by ilausuch 2 months ago

About the message

Mojolicious::Plugin::AssetPack::Pipe::Sass requires ruby. https://ruby-lang.org/en/documentation/installation

The url is wrong, should be
https://www.ruby-lang.org/en/documentation/installation/

#10 Updated by ilausuch 2 months ago

https://github.com/os-autoinst/openQA/pull/4050
I am preparing a test as a script in tools to try to reproduce this locally and to have a sooner than in openqa and before merge

#11 Updated by ilausuch 2 months ago

Now the PR is merged. Waiting for some completed tests in https://openqa.opensuse.org/group_overview/24

#12 Updated by ilausuch 2 months ago

  • Subject changed from [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" size:M to [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Mojolicious::Plugin::AssetPack::Pipe::Sass requires"
  • Status changed from In Progress to Resolved

I changed the original title of this ticket to reflect the real reason of the failure.
And this ticket now is resolved and checked that since the PR was merged (we had more than 24 hours of testing without failures with this error) see: https://openqa.opensuse.org/group_overview/24
However the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" still appears on the logs but it is covered by the ticket #95768

Also available in: Atom PDF