[qam][sle][opensuse][functional][x11regressions]test fails in firefox with what looks like temporary problem to access html5test.com
#2 Updated by asmorodskyi over 5 years ago
Behavior changed a little bit but still looks like same problem
#3 Updated by okurz over 5 years ago
- Assignee deleted (
- Priority changed from Normal to Low
don't know what should be done. For now no need to change anything I assume. If we are annoyed often enough we can try some alternatives, e.g. try three alternatives: html5test.com, something else and something ... more else.
#4 Updated by asmorodskyi over 5 years ago
- Priority changed from Low to Normal
#10 Updated by okurz over 4 years ago
- Subject changed from test fails in firefox with what looks like temporary problem to access html5test.com to [qam][sle][opensuse][functional][x11regressions]test fails in firefox with what looks like temporary problem to access html5test.com
- Category changed from Bugs in existing tests to Enhancement to existing tests
- Status changed from Feedback to In Progress
- Assignee set to dasantiago
- Priority changed from Low to Normal
Incorporating content from duplicate #20228:
Relying on external webservers is not optimal. Even visiting the same webpage (in that case html5test.com) has been proven to be quite unstable, since within 24h timeframe, their server triggers different behavior of Firefox' Tracking Protection mechanism. As a result, yesterday, the same test passed, but today is failing. Also, re-adjusting the needle is not a option, since by adding the shield icon, the URL gets shifted to the more right direction (thus the needle with missmatch).
As a solution, I would propose to replace 'html5test.com' with a local webpage generated by a webserver. Some approaches:
zypper in -t pattern lamp_server (this will also install 'apache2-example-pages' pkg which will automatically create an index.html accessible by localhost)
systemctl start apache2
curl -s localhost | grep "It works!"
// Now that you know that your page is accessible, fire Firefox and do your tests
Alternatively, we could build a docker image that provides a pre-configured website by default:
zypper in docker
Another approach suggested by asmorodskyi would be to use https://www.lighttpd.net/ . The benefit is that in this approach we will not rely on zypper to install apache2 but just rar.gz as test resource. In that case we are not adding additional reason to fail the test.
Relying on https://html5test.com proved to be pretty bad once again, as currently every single firefox/chromium test fails because of an expires SSL certificate.
I'll make a PR shortly that uses http in those tests, but that will require new needles due to the URL change (and changed security)...
For a long-term solution I'd suggest adding a static html page somewhere on https://www.opensuse.org, which should be reachable from everywhere and be reliable enough, at least we can fix it ourselves if it breaks.
[favogt] Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?
[dasantiago] IMO it should check the features of the browser, that it's built against all required libs to support WebGL, html5 video and audio etc.
[favogt] Does it need to have SSL?
The best solution, imho, is to have a local server serving the pages. Depending on what are the requirements for the static page, apache might be overkill...
while true; do echo -e "HTTP/1.1 200 OK\n\n Hello $(whoami)! Now is $(date)" | nc -l localhost 8080; done
enough or is it required a more complex case?
[dasantiago] Yes, it needs to check that the entire cert chain is valid. That is why I suggested https://www.opensuse.org. That rules out starting a local server though, as testing self-signed certs misses the point...
The website code is on github. I will follow this with infra team, so we can have it hosted.
After talking to the infra team they suggested two hosts:
We can set up our html5 page there.
#12 Updated by pcervinka over 4 years ago
Another occurrence across all firefox test in openqa, html5test.com is not responding:
#15 Updated by okurz over 4 years ago
- Assignee changed from dasantiago to okurz
#17 Updated by okurz over 4 years ago
- Status changed from In Progress to Resolved
All merged and SLE needles created. I checked many maintenance jobs, could be some will still fail because they have been triggered before the needles have been created. But it can work in general, see staging job: https://openqa.suse.de/tests/1411129#step/firefox/9
#19 Updated by okurz over 4 years ago
I created one more needle for SLE12SP1 where I could find accordingly failed jobs but https://openqa.suse.de/tests/1411879#step/firefox_smoke/7 is a different problem. The update notifier covers the title bar and therefore the needle is not recognized. That is something else. IMHO the needle design is not resilient. I would not try to needle in that area :) but I am also wondering where should be updates at all. They are supposed to be all installed at this time, or not?