Project

General

Profile

action #14848

[qam][sle][opensuse][functional][x11regressions]test fails in firefox with what looks like temporary problem to access html5test.com

Added by okurz over 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Target version:
-
Start date:
2016-11-16
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-12-SP3-Server-DVD-x86_64-xen@64bit fails in
firefox

Reproducible

Fails since (at least) Build 0057 (current job)

Expected result

Last good: 0057 (or more recent)

Further details

Always latest result in this scenario: latest


Related issues

Related to openQA Tests - action #25972: [opensuse][krypton][functional][medium] test fails in firefox - html5test page does not show up, is the address typed at all?Resolved2017-10-112018-02-13

Has duplicate openQA Tests - action #26976: [qam] test fails in firefox - http://html5test.com/ brokenRejected2017-10-24

Has duplicate openQA Tests - action #20228: [qam]stop relying on html5test.comRejected2017-07-03

History

#1 Updated by okurz over 5 years ago

retriggered without changes: https://openqa.suse.de/tests/637438
if it just happened once we will probably not do anything about it.

#2 Updated by asmorodskyi over 5 years ago

Behavior changed a little bit but still looks like same problem
https://openqa.suse.de/tests/640738#step/firefox/4

#3 Updated by okurz over 5 years ago

  • Assignee deleted (okurz)
  • Priority changed from Normal to Low

don't know what should be done. For now no need to change anything I assume. If we are annoyed often enough we can try some alternatives, e.g. try three alternatives: html5test.com, something else and something ... more else.

#4 Updated by asmorodskyi over 5 years ago

  • Priority changed from Low to Normal

https://openqa.suse.de/tests/692068
https://openqa.suse.de/tests/692069
https://openqa.suse.de/tests/691932
https://openqa.suse.de/tests/691930

Fresh examples of failure , increase priority back to normal . Best would be setup our local instance of html5test or find another more stable provider

#5 Updated by asmorodskyi over 5 years ago

around 20 other tests failing , looks like html5test is down , issue reproduces in my local chrome instance so could not be browser bug

#6 Updated by asmorodskyi over 5 years ago

  • Status changed from In Progress to Feedback
  • Priority changed from Normal to Low

Issue not reproduces anymore return it back to Low

#7 Updated by okurz over 4 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: lvm
https://openqa.suse.de/tests/1245344

#8 Updated by okurz over 4 years ago

  • Related to action #25972: [opensuse][krypton][functional][medium] test fails in firefox - html5test page does not show up, is the address typed at all? added

#9 Updated by okurz over 4 years ago

  • Has duplicate action #26976: [qam] test fails in firefox - http://html5test.com/ broken added

#10 Updated by okurz over 4 years ago

  • Subject changed from test fails in firefox with what looks like temporary problem to access html5test.com to [qam][sle][opensuse][functional][x11regressions]test fails in firefox with what looks like temporary problem to access html5test.com
  • Category changed from Bugs in existing tests to Enhancement to existing tests
  • Status changed from Feedback to In Progress
  • Assignee set to dasantiago
  • Priority changed from Low to Normal

Incorporating content from duplicate #20228:

Relying on external webservers is not optimal. Even visiting the same webpage (in that case html5test.com) has been proven to be quite unstable, since within 24h timeframe, their server triggers different behavior of Firefox' Tracking Protection mechanism. As a result, yesterday, the same test passed, but today is failing. Also, re-adjusting the needle is not a option, since by adding the shield icon, the URL gets shifted to the more right direction (thus the needle with missmatch).

As a solution, I would propose to replace 'html5test.com' with a local webpage generated by a webserver. Some approaches:

zypper in -t pattern lamp_server (this will also install 'apache2-example-pages' pkg which will automatically create an index.html accessible by localhost)
systemctl start apache2
curl -s localhost | grep "It works!"
// Now that you know that your page is accessible, fire Firefox and do your tests

Alternatively, we could build a docker image that provides a pre-configured website by default:
zypper in docker
docker run

Another approach suggested by asmorodskyi would be to use https://www.lighttpd.net/ . The benefit is that in this approach we will not rely on zypper to install apache2 but just rar.gz as test resource. In that case we are not adding additional reason to fail the test.

dasantiago:
Relying on https://html5test.com proved to be pretty bad once again, as currently every single firefox/chromium test fails because of an expires SSL certificate.
I'll make a PR shortly that uses http in those tests, but that will require new needles due to the URL change (and changed security)...
For a long-term solution I'd suggest adding a static html page somewhere on https://www.opensuse.org, which should be reachable from everywhere and be reliable enough, at least we can fix it ourselves if it breaks.

[favogt] Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?

[dasantiago] IMO it should check the features of the browser, that it's built against all required libs to support WebGL, html5 video and audio etc.

[favogt] Does it need to have SSL?
The best solution, imho, is to have a local server serving the pages. Depending on what are the requirements for the static page, apache might be overkill...
Is

while true; do echo -e "HTTP/1.1 200 OK\n\n Hello $(whoami)! Now is $(date)" | nc -l localhost 8080; done

enough or is it required a more complex case?

[dasantiago] Yes, it needs to check that the entire cert chain is valid. That is why I suggested https://www.opensuse.org. That rules out starting a local server though, as testing self-signed certs misses the point...

dasantiago:
The website code is on github. I will follow this with infra team, so we can have it hosted.
After talking to the infra team they suggested two hosts:

conncheck.opensuse.org
static.opensuse.org

We can set up our html5 page there.

#11 Updated by okurz over 4 years ago

  • Has duplicate action #20228: [qam]stop relying on html5test.com added

#12 Updated by pcervinka over 4 years ago

Another occurrence across all firefox test in openqa, html5test.com is not responding:
https://openqa.suse.de/tests/1386322#step/firefox/11

#13 Updated by SLindoMansilla over 4 years ago

What about implementing a soft-fail here when html5test is not responding?

This issue is causing a lot of false positives: https://openqa.suse.de/tests/1395757

#14 Updated by okurz over 4 years ago

https://progress.opensuse.org/issues/21856 has been resolved. I think we can use static.opensuse.org now instead. I have a patch prepared and testing that locally to rewrite all (both) occassions of html5test.com

#15 Updated by okurz over 4 years ago

  • Assignee changed from dasantiago to okurz

#16 Updated by okurz over 4 years ago

merged, checking SLE needles, e.g. https://openqa.suse.de/tests/1409518

#17 Updated by okurz over 4 years ago

  • Status changed from In Progress to Resolved

All merged and SLE needles created. I checked many maintenance jobs, could be some will still fail because they have been triggered before the needles have been created. But it can work in general, see staging job: https://openqa.suse.de/tests/1411129#step/firefox/9

#18 Updated by osukup over 4 years ago

looks like missing needles for SLE12GA and SLE12SP1 ..

#19 Updated by okurz over 4 years ago

I created one more needle for SLE12SP1 where I could find accordingly failed jobs but https://openqa.suse.de/tests/1411879#step/firefox_smoke/7 is a different problem. The update notifier covers the title bar and therefore the needle is not recognized. That is something else. IMHO the needle design is not resilient. I would not try to needle in that area :) but I am also wondering where should be updates at all. They are supposed to be all installed at this time, or not?

Also available in: Atom PDF