Project

General

Profile

Actions

action #162311

closed

coordination #154768: [saga][ux] State-of-art user experience for openQA

coordination #157510: [epic] Up-to-date JavaScript stack

[sporadic] openqa-in-openqa-test fails in openqa_webui due to CSS files not successfully generated size:M

Added by tinita about 1 month ago. Updated 22 days ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2024-06-14
Due date:
% Done:

0%

Estimated time:

Description

Observation

openQA test in scenario openqa-Tumbleweed-dev-x86_64-openqa_from_git@64bit-2G fails in
openqa_webui

Acceptance criteria

  • AC1: The CSS files of the openQA web UI are generated reliably (without sometimes running into SASS-related errors).

Reproducible

Fails since (at least) Build :TW.29212

Expected result

Last good: :TW.29211 (or more recent)

Suggestions

  • DONE Understand why openQA-in-openQA tests show broken CSS sometimes -> Because sass failed to build assets and AssetPack is ignoring errors so our packages in devel:openQA have sometimes broken CSS since the bootstrap 4->5 migration
  • Provide a fix for how sass builds CSS to the according package https://build.opensuse.org/request/show/1181472
  • Submit the according patch temporarily also into devel:openQA with a link back to this ticket
  • Await the acceptance of the according SR into Factory, verify that it's effective for openQA, remove the link in devel:openQA again

Further details

Always latest result in this scenario: latest

Out of scope

  • Future-proof solution: #162500
  • AssetPack should fail if sass fails #162503

Related issues 6 (3 open3 closed)

Related to openQA Tests - action #138362: [opensuse] test fails in openqa_webui: Apache server not listening at 127.0.0.1Resolved2023-10-23

Actions
Related to openQA Tests - action #64523: [openqa-in-openqa][sporadic] test fails in sometimes openqa_webui waiting for output from nohup.out, potential race condition?New2020-03-16

Actions
Related to openQA Project - action #137426: test fails in openqa_webui auto_review:"Test died:.*Listening at.*timed out"Resolvedokurz2023-10-04

Actions
Blocks openQA Project - action #162242: openqa-in-openqa test fails in dashboardResolvedmkittler2024-06-13

Actions
Copied to openQA Project - action #162500: Switch to more reliable CSS generation, e.g. using sasscNew

Actions
Copied to openQA Project - action #162503: AssetPack should fail if sass fails size:MWorkable

Actions
Actions #1

Updated by tinita about 1 month ago

  • Subject changed from openqa-in-openqa-test fails in openqa_webui to [sporadic] openqa-in-openqa-test fails in openqa_webui
Actions #2

Updated by tinita about 1 month ago

nohop.out contains this:
https://openqa.opensuse.org/tests/4273694/logfile?filename=openqa_webui-openqa_nohup_out.txt

TypeError: String can't be coerced into Integer
  Use --trace for backtrace.
[info] Listening at "http://127.0.0.1:9526"
Web application available at http://127.0.0.1:9526
[info] Listening at "http://[::1]:9526"

and grep -qP "Listening at.*(127.0.0.1|localhost)" <(tail -F -n0 nohup.out) fails.
I have been looking at this code before. Why does it use a -n0 here?

Actions #3

Updated by nicksinger about 1 month ago

tinita wrote in #note-2:

nohop.out contains this:
https://openqa.opensuse.org/tests/4273694/logfile?filename=openqa_webui-openqa_nohup_out.txt

TypeError: String can't be coerced into Integer
  Use --trace for backtrace.
[info] Listening at "http://127.0.0.1:9526"
Web application available at http://127.0.0.1:9526
[info] Listening at "http://[::1]:9526"

and grep -qP "Listening at.*(127.0.0.1|localhost)" <(tail -F -n0 nohup.out) fails.
I have been looking at this code before. Why does it use a -n0 here?

I think the -n0 mainly prevents tail to print something if nohup.out is already present.

Actions #4

Updated by tinita about 1 month ago

It's hard to say what happens in the successful case as we don't upload the log there.
That could be something one could do in an investigation.

Actions #5

Updated by tinita about 1 month ago

  • Status changed from New to In Progress
  • Assignee set to tinita
Actions #6

Updated by tinita about 1 month ago

I just ran a test with an addeed die after the grep, so that logs get uploaded.
In this case the log contains:

[info] Listening at "http://127.0.0.1:9526"
Web application available at http://127.0.0.1:9526
[info] Listening at "http://[::1]:9526"

So the different content seems to cause the grep to fail. Although I don't know why.

TypeError: String can't be coerced into Integer
  Use --trace for backtrace.

That message comes from ruby apparently.

Actions #7

Updated by tinita about 1 month ago

The openqa_webui-openqa_services.log looks the same in both cases:

Jun 14 10:59:21 susetest systemd[1]: Starting The Apache Webserver...
Jun 14 10:59:21 susetest start_apache2[25488]: AH00112: Warning: DocumentRoot [/usr/share/openqa/public] does not exist
Jun 14 10:59:21 susetest start_apache2[25488]: AH00557: httpd-prefork: apr_sockaddr_info_get() failed for susetest
Jun 14 10:59:21 susetest start_apache2[25488]: AH00558: httpd-prefork: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
Jun 14 10:59:21 susetest systemd[1]: Started The Apache Webserver.
Actions #8

Updated by tinita about 1 month ago

nicksinger wrote in #note-3:

I think the -n0 mainly prevents tail to print something if nohup.out is already present.

Yeah, but that wouldn't actually hurt.
I think in this case the log is already there with the expected content when the grep starts, so it will never get the expected input.
It's just weird that it seems to be too slow only in the case we get the ruby error message.
Will debug further.

Actions #9

Updated by tinita about 1 month ago

  • Related to action #138362: [opensuse] test fails in openqa_webui: Apache server not listening at 127.0.0.1 added
Actions #10

Updated by tinita about 1 month ago

  • Related to action #64523: [openqa-in-openqa][sporadic] test fails in sometimes openqa_webui waiting for output from nohup.out, potential race condition? added
Actions #11

Updated by tinita about 1 month ago

  • Related to action #137426: test fails in openqa_webui auto_review:"Test died:.*Listening at.*timed out" added
Actions #12

Updated by tinita about 1 month ago

https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/179 Improve waiting for webserver

This can at least help with debugging.

Actions #13

Updated by tinita about 1 month ago

It seems to be consistent (but still monitoring):
When the TypeError happens
https://openqa.opensuse.org/tests/4273783#step/openqa_webui/31
then also the CSS seems to be missing:
https://openqa.opensuse.org/tests/4273783#step/dashboard/5

Actions #14

Updated by openqa_review about 1 month ago

  • Due date set to 2024-06-29

Setting due date based on mean cycle time of SUSE QE Tools

Actions #17

Updated by tinita about 1 month ago

Actions #18

Updated by mkittler about 1 month ago

  • Related to action #162242: openqa-in-openqa test fails in dashboard added
Actions #19

Updated by mkittler about 1 month ago

For the record: I have seen this before on my local machine during development. One time I didn't do any changes except for adding the --trace parameter (like in https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/180) but I haven't seen it since I added the parameter so I cannot share a back-trace. I'll keep the --trace parameter in my local setup and will write a comment if I see it again.

Actions #20

Updated by tinita about 1 month ago

  • Status changed from In Progress to Blocked

Yeah, It looks like the same as #162242

https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/182 Add comments about temporary sass patch

I created https://build.opensuse.org/request/show/1181342

Waiting for that until I can link it in devel:openQA

If the issue happens again we can look at the stacktrace.

Actions #21

Updated by tinita about 1 month ago · Edited

Ok, we got a failure:
https://openqa.opensuse.org/tests/4276734/logfile?filename=dashboard-openqa_nohup_out.txt

/usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/value/number.rb:409:in `-': nil can't be coerced into Integer (TypeError)

      (num1 - num2).abs < epsilon
              ^^^^
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/value/number.rb:409:in `basically_equal?'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/value/number.rb:403:in `basically_equal?'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/value/number.rb:226:in `eql?'
    from /usr/lib64/ruby/3.3.0/set.rb:394:in `include?'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:54:in `block in _perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:52:in `map'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:52:in `_perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/node.rb:50:in `perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:53:in `block in _perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:52:in `map'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:52:in `_perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/node.rb:50:in `perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:53:in `block in _perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:52:in `map'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/map_literal.rb:52:in `_perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/node.rb:50:in `perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/funcall.rb:182:in `perform_arg'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/funcall.rb:129:in `block in _perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/funcall.rb:129:in `each'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/funcall.rb:129:in `each_with_index'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/funcall.rb:129:in `each'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/funcall.rb:129:in `map'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/funcall.rb:129:in `_perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/tree/node.rb:50:in `perform'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:478:in `visit_variable'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `block in visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `block in with_base'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:135:in `with_frame'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `with_base'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:325:in `block (2 levels) in visit_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:325:in `map'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:325:in `block in visit_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:88:in `block in with_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:135:in `with_frame'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:88:in `with_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:322:in `visit_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `block in visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `block in with_base'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:135:in `with_frame'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `with_base'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:325:in `block (2 levels) in visit_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:325:in `map'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:325:in `block in visit_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:88:in `block in with_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:135:in `with_frame'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:88:in `with_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:322:in `visit_import'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `block in visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `block in with_base'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:135:in `with_frame'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/stack.rb:79:in `with_base'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:158:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `block in visit_children'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `map'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:52:in `visit_children'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:167:in `block in visit_children'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:179:in `with_environment'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:166:in `visit_children'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `block in visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:186:in `visit_root'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/base.rb:36:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:157:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/visitors/perform.rb:10:in `visit'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/root_node.rb:36:in `css_tree'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/tree/root_node.rb:20:in `render'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/engine.rb:290:in `render'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/exec/sass_scss.rb:396:in `run'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/exec/sass_scss.rb:63:in `process_result'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/exec/base.rb:50:in `parse'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/exec/base.rb:18:in `parse!'
    from /usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/bin/sass:13:in `<top (required)>'
    from /usr/bin/sass:25:in `load'
    from /usr/bin/sass:25:in `<main>'
[info] Listening at "http://127.0.0.1:9526"
Web application available at http://127.0.0.1:9526
[info] Listening at "http://[::1]:9526"

https://github.com/sass/dart-sass/issues/2045 looks like the same issue, but I don't understand the explanation yet

Actions #22

Updated by tinita about 1 month ago · Edited

I am able to reproduce it sometimes with:

% cat in
@import "../node_modules/bootstrap/scss/bootstrap.scss";

% sass -s --trace -I /tmp/install/openQA/script/../assets/stylesheets -I /tmp/install/openQA/script/../assets --scss <in >out
/usr/lib64/ruby/gems/3.3.0/gems/sass-3.7.4/lib/sass/script/value/number.rb:409:in `-': nil can't be coerced into Integer (TypeError)
...
Actions #23

Updated by tinita about 1 month ago

  • Status changed from Blocked to In Progress

The arguments to the failing basically_equal call are: 3, first

Actions #24

Updated by tinita about 1 month ago

Two problematic snippets seem to be:

// node_modules/bootstrap/scss/_variables.scss
// scss-docs-start zindex-levels-map
$zindex-levels: (
  n1: -1, // <------------------
  0: 0,
  1: 1,
  2: 2,
  3: 3
) !default;
// node_modules/bootstrap/scss/_utilities.scss
    "order": (
      responsive: true,
      property: order,
      values: (
        first: -1, // <------------
        0: 0,
        1: 1,
        2: 2,
        3: 3,
        4: 4,
        5: 5,
        last: 6,
      ),

not sure why it only happens sometimes.

Actions #25

Updated by tinita about 1 month ago

I guess there are several options now. I could try a patch that would allow to also compare strings.
But this gem is deprecated, and apparently there are alternatives, but it's not clear which would be recommended.
There's sassc which uses libsass. Of course it has completely different commandline options.
Then there is dartsass, and 3 rubygems which are wrappers around dartsass. But none of them in openSUSE.
Also 4 different npm packages that do sass.

Sooner or later we will have to do something about it.
I'll try the patch first.

Actions #26

Updated by tinita about 1 month ago

I created a patch: https://build.opensuse.org/request/show/1181472
Will now shortly look into sassc, whether it might be an option.

Actions #27

Updated by tinita about 1 month ago

sassc looks good at the first glance. At least the openqa-in-openqa dashboard test runs fine. There are minor diffs in the generated scss files, but they seem to be cosmetic.
Of course for using that we would have to patch AssetPack as well.

Actions #29

Updated by mkittler about 1 month ago

Nice that you could get to the bottom of this. I guess it makes sense to switch to sassc then. For the time being we can rely on https://build.opensuse.org/request/show/1181472.

Actions #30

Updated by mkittler about 1 month ago

  • Related to deleted (action #162242: openqa-in-openqa test fails in dashboard)
Actions #31

Updated by mkittler about 1 month ago

  • Blocks action #162242: openqa-in-openqa test fails in dashboard added
Actions #32

Updated by okurz about 1 month ago

  • Copied to action #162500: Switch to more reliable CSS generation, e.g. using sassc added
Actions #33

Updated by okurz about 1 month ago

  • Copied to action #162503: AssetPack should fail if sass fails size:M added
Actions #34

Updated by okurz about 1 month ago

  • Subject changed from [sporadic] openqa-in-openqa-test fails in openqa_webui to [sporadic] openqa-in-openqa-test fails in openqa_webui due to CSS files not successfully generated
  • Description updated (diff)
Actions #35

Updated by okurz about 1 month ago

  • Subject changed from [sporadic] openqa-in-openqa-test fails in openqa_webui due to CSS files not successfully generated to [sporadic] openqa-in-openqa-test fails in openqa_webui due to CSS files not successfully generated size:M
Actions #36

Updated by okurz about 1 month ago

  • Parent task set to #157510
Actions #37

Updated by mkittler about 1 month ago

Submit the according patch temporarily also into devel:openQA with a link back to this ticket

I created https://build.opensuse.org/package/show/devel:openQA/rubygem-sass via osc copypac home:tinita:branches:devel:languages:ruby:extensions rubygem-sass devel:openQA.

Actions #38

Updated by tinita about 1 month ago

  • Status changed from In Progress to Blocked
Actions #40

Updated by tinita about 1 month ago

https://build.opensuse.org/request/show/1182090 was accepted.
Now I guess I can link it in the subprojects of devel:openQA?

Actions #41

Updated by okurz about 1 month ago

you mean to overwrite the copypac version by mkittler with a linkpac version? Yes, please

Actions #42

Updated by okurz 25 days ago

  • Due date deleted (2024-06-29)
  • Status changed from Blocked to Workable

As https://build.opensuse.org/request/show/1182090 was accepted days ago please continue with the next step

Actions #43

Updated by tinita 23 days ago

  • Status changed from Workable to In Progress
Actions #44

Updated by tinita 23 days ago

  • Status changed from In Progress to Feedback

I saw that Marius already did linkpac for Leap 15.4/15.5/15.6.
I just removed https://build.opensuse.org/package/show/devel:openQA/rubygem-sass now.

Actions #45

Updated by tinita 22 days ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF