Project

General

Profile

Actions

action #95179

closed

[sporadic] containers: eventually the tests fails on single_container_webui step with the error "Mojolicious::Plugin::AssetPack::Pipe::Sass requires"

Added by ilausuch over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2021-07-07
Due date:
2021-07-28
% Done:

0%

Estimated time:

Description

Motivation

Eventually we saw this error in the container tests.

Can not create MOJO_TMPDIR : mkdir /var/lib/openqa/share/factory/tmp: Permission denied at /usr/share/openqa/script/../lib/OpenQA/Setup.pm line 316.

https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=openqa&flavor=dev&machine=64bit-2G&test=openqa_from_containers&version=Tumbleweed

Acceptance Criteria

  • AC 1: The test is fixed and doesn't break here

Steps to reproduce

Suggestions

  • Try to reproduce this can locally
  • Try to fix it in openQA itself not just a package
  • The same menssage is logged by two different components, maybe is a concurrency problem
Actions #1

Updated by ilausuch over 2 years ago

  • Subject changed from cotainers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" to containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp"
Actions #2

Updated by okurz over 2 years ago

  • Target version set to Ready
Actions #3

Updated by ilausuch over 2 years ago

  • Subject changed from containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" to [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #4

Updated by ilausuch over 2 years ago

  • Status changed from Workable to In Progress
  • Assignee set to ilausuch
Actions #5

Updated by ilausuch over 2 years ago

Working on the investigation, I found something that could be only circumstantial but maybe not.

The run_openqa.sh before we adapt that to have two versions (one for all services together and other to have this split and ready for HA) didn’t execute livehandle (see https://github.com/os-autoinst/openQA/commit/e4e4d4973b2dcc2e02ae136da91d688ea6417c8d#diff-601001db02292ee9c9a3c7f4dbf8699b9465ee448277cef81ecebd2bb83b7a58)

start_daemon -u geekotest /usr/share/openqa/script/openqa-scheduler &
start_daemon -u geekotest /usr/share/openqa/script/openqa-websockets &
start_daemon -u geekotest /usr/share/openqa/script/openqa gru -m production run &
apache2ctl start
start_daemon -u geekotest /usr/share/openqa/script/openqa prefork -m production –proxy

And now this runs the livehandler also

start_daemon -u geekotest /usr/share/openqa/script/openqa-scheduler &
start_daemon -u geekotest /usr/share/openqa/script/openqa-websockets &
start_daemon -u geekotest /usr/share/openqa/script/openqa-livehandler &
start_daemon -u geekotest /usr/share/openqa/script/openqa gru -m production run &
apache2ctl start
start_daemon -u geekotest /usr/share/openqa/script/openqa prefork -m production --proxy

At the same time we can see in the errors of https://openqa.opensuse.org/tests/1837881#step/single_container_webui/108

# Test died: output not validating at openqa/lib/utils.pm line 101.

This line (https://github.com/os-autoinst/openQA/blob/1248f9f088268759cbed5c91104c7ddb29d8c9d8/lib/OpenQA/Utils.pm#L101) refers to a this piece of code

# returns the url to the web socket proxy started via openqa-livehandler
sub determine_web_ui_web_socket_url {
my ($job_id) = @_;
return "liveviewhandler/tests/$job_id/developer/ws-proxy";
}

So this could be a coincidence or not

Actions #6

Updated by openqa_review over 2 years ago

  • Due date set to 2021-07-28

Setting due date based on mean cycle time of SUSE QE Tools

Actions #7

Updated by mkittler over 2 years ago

The directory is only created by the web UI but not other services like the livehandler. So I don't think the introduction of livehandler makes a difference.

More importantly, I'd like to note that "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" is not the reason why the test fails. It is just a log message which even appears on successful runs: https://openqa.opensuse.org/tests/1839850#step/single_container_webui/8
(The same counts for AH00558: httpd-prefork: Could not reliably determine the server's fully qualified domain name, using 172.18.0.3. Set the 'ServerName' directive globally to suppress this message.)

The last log message is Mojolicious::Plugin::AssetPack::Pipe::Sass requires ruby. https://ruby-lang.org/en/documentation/installation which is also absent in successful runs. So maybe that's the culprit? Of course it leaves the question why ruby is attempted to be used in some cases. I would assume it is only a build/development requirement as the asset cache is supplied by the rpm package.

Actions #8

Updated by ilausuch over 2 years ago

For now I created this PR to ensure that the permissions are good for geekotest user
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/71

I tested locally that I the permissions were incorrect and was imposible to create these directories. But I don't know how in openqa only eventually this failed. So independent that is the real problem or not here, I think this solution could help if in the future we test more deeply

Now I am going to investigate your suggestions

Actions #9

Updated by ilausuch over 2 years ago

About the message

Mojolicious::Plugin::AssetPack::Pipe::Sass requires ruby. https://ruby-lang.org/en/documentation/installation

The url is wrong, should be
https://www.ruby-lang.org/en/documentation/installation/

Actions #10

Updated by ilausuch over 2 years ago

https://github.com/os-autoinst/openQA/pull/4050
I am preparing a test as a script in tools to try to reproduce this locally and to have a sooner than in openqa and before merge

Actions #11

Updated by ilausuch over 2 years ago

Now the PR is merged. Waiting for some completed tests in https://openqa.opensuse.org/group_overview/24

Actions #12

Updated by ilausuch over 2 years ago

  • Subject changed from [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" size:M to [sporadic] containers: eventually the tests fails on single_container_webui step with the error "Mojolicious::Plugin::AssetPack::Pipe::Sass requires"
  • Status changed from In Progress to Resolved

I changed the original title of this ticket to reflect the real reason of the failure.
And this ticket now is resolved and checked that since the PR was merged (we had more than 24 hours of testing without failures with this error) see: https://openqa.opensuse.org/group_overview/24
However the error "Permission denied" when tries to create the directory "/var/lib/openqa/share/factory/tmp" still appears on the logs but it is covered by the ticket #95768

Actions

Also available in: Atom PDF