action #20228
closed[qam]stop relying on html5test.com
0%
Description
Observation¶
openQA test in scenario sle-12-SP1-Server-DVD-Updates-x86_64-qam-gnome@64bit fails in
firefox
Relying on external webservers is not optimal. Even visiting the same webpage (in that case html5test.com) has been proven to be quite unstable, since within 24h timeframe, their server triggers different behavior of Firefox' Tracking Protection mechanism. As a result, yesterday, the same test passed, but today is failing. Also, re-adjusting the needle is not a option, since by adding the shield icon, the URL gets shifted to the more right direction (thus the needle with missmatch).
As a solution, I would propose to replace 'html5test.com' with a local webpage generated by a webserver. Some approaches:
zypper in -t pattern lamp_server (this will also install 'apache2-example-pages' pkg which will automatically create an index.html accessible by localhost)
systemctl start apache2
curl -s localhost | grep "It works!"
// Now that you know that your page is accessible, fire Firefox and do your tests
Alternatively, we could build a docker image that provides a pre-configured website by default:
zypper in docker
docker run
Another approach suggested by @asmorodskyi would be to use https://www.lighttpd.net/ . The benefit is that in this approach we will not rely on zypper to install apache2 but just rar.gz as test resource. In that case we are not adding additional reason to fail the test.
What do you think?
Reproducible¶
Fails since (at least) Build 20170703-1 (current job)
Expected result¶
Last good: 20170702-1 (or more recent)
Further details¶
Always latest result in this scenario: latest
Updated by SLindoMansilla almost 7 years ago
- Category changed from Bugs in existing tests to Enhancement to existing tests
Good point it would be a good enhancement to the test. But it is not a bug in the test.
Updated by okurz almost 7 years ago
- Subject changed from stop relying on html5test.com to [qam]stop relying on html5test.com
Updated by favogt over 6 years ago
- Priority changed from Low to Normal
Relying on https://html5test.com proved to be pretty bad once again, as currently every single firefox/chromium test fails because of an expires SSL certificate.
I'll make a PR shortly that uses http in those tests, but that will require new needles due to the URL change (and changed security)...
For a long-term solution I'd suggest adding a static html page somewhere on https://www.opensuse.org, which should be reachable from everywhere and be reliable enough, at least we can fix it ourselves if it breaks.
Updated by dasantiago over 6 years ago
Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?
Does it need to have SSL?
The best solution, imho, is to have a local server serving the pages. Depending on what are the requirements for the static page, apache might be overkill...
Is
while true; do echo -e "HTTP/1.1 200 OK\n\n Hello $(whoami)! Now is $(date)" | nc -l localhost 8080; done
enough or is it required a more complex case?
Updated by favogt over 6 years ago
Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?
IMO it should check the features of the browser, that it's built against all required libs to support WebGL, html5 video and audio etc.
Does it need to have SSL?
Yes, it needs to check that the entire cert chain is valid. That is why I suggested https://www.opensuse.org.
That rules out starting a local server though, as testing self-signed certs misses the point...
Currently html5test.com is down again as api.whichbrowser.net (which had the expired cert) now completely refuses to respond and all openqa tests will once again fail...
Updated by dasantiago over 6 years ago
favogt wrote:
Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?
IMO it should check the features of the browser, that it's built against all required libs to support WebGL, html5 video and audio etc.
Does it need to have SSL?
Yes, it needs to check that the entire cert chain is valid. That is why I suggested https://www.opensuse.org.
That rules out starting a local server though, as testing self-signed certs misses the point...Currently html5test.com is down again as api.whichbrowser.net (which had the expired cert) now completely refuses to respond and all openqa tests will once again fail...
The website code is on github. I will follow this with infra team, so we can have it hosted.
Updated by favogt over 6 years ago
Note that not html5test.com itself was down, but some of the third-party APIs it uses.
In this case, api.whichbrowser.net (also on GitHub).
Updated by dasantiago over 6 years ago
After talking to the infra team they suggested two hosts:
- conncheck.opensuse.org
- static.opensuse.org
We can set up our html5 page there.
Updated by dasantiago over 6 years ago
unfortunately, both html5test is not a static page :-(
https://github.com/WebPlatformTest/HTML5test/blob/version-8.0/backend/index.php#L60
Updated by vsvecova over 6 years ago
Same issue has been happening again for the past ~3 days.
Examples:
https://openqa.suse.de/tests/1351200#step/firefox/10
https://openqa.suse.de/tests/1351325#step/firefox/10
https://openqa.suse.de/tests/1351146#step/firefox/10
Looks like the problem has been coming back periodically over the past year or so; cf. closed older ticket poo#12282.
Btw. there is also another recent duplicate ticket poo#26976.
Updated by okurz over 6 years ago
- Status changed from New to Rejected
incorporated all content into duplicatee #14848
Updated by okurz over 6 years ago
- Is duplicate of action #14848: [qam][sle][opensuse][functional][x11regressions]test fails in firefox with what looks like temporary problem to access html5test.com added