QA » openQA Project » openQA Tests

don't know what should be done. For now no need to change anything I assume. If we are annoyed often enough we can try some alternatives, e.g. try three alternatives: html5test.com, something else and something ... more else.

https://openqa.suse.de/tests/692068
https://openqa.suse.de/tests/692069
https://openqa.suse.de/tests/691932
https://openqa.suse.de/tests/691930

Fresh examples of failure , increase priority back to normal . Best would be setup our local instance of html5test or find another more stable provider

Issue not reproduces anymore return it back to Low

Incorporating content from duplicate #20228:

Relying on external webservers is not optimal. Even visiting the same webpage (in that case html5test.com) has been proven to be quite unstable, since within 24h timeframe, their server triggers different behavior of Firefox' Tracking Protection mechanism. As a result, yesterday, the same test passed, but today is failing. Also, re-adjusting the needle is not a option, since by adding the shield icon, the URL gets shifted to the more right direction (thus the needle with missmatch).

As a solution, I would propose to replace 'html5test.com' with a local webpage generated by a webserver. Some approaches:

zypper in -t pattern lamp_server (this will also install 'apache2-example-pages' pkg which will automatically create an index.html accessible by localhost)
systemctl start apache2
curl -s localhost | grep "It works!"
// Now that you know that your page is accessible, fire Firefox and do your tests

Alternatively, we could build a docker image that provides a pre-configured website by default:
zypper in docker
docker run

Another approach suggested by @asmorodskyi would be to use https://www.lighttpd.net/ . The benefit is that in this approach we will not rely on zypper to install apache2 but just rar.gz as test resource. In that case we are not adding additional reason to fail the test.

dasantiago:
Relying on https://html5test.com proved to be pretty bad once again, as currently every single firefox/chromium test fails because of an expires SSL certificate.
I'll make a PR shortly that uses http in those tests, but that will require new needles due to the URL change (and changed security)...
For a long-term solution I'd suggest adding a static html page somewhere on https://www.opensuse.org, which should be reachable from everywhere and be reliable enough, at least we can fix it ourselves if it breaks.

[favogt] Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?

[dasantiago] IMO it should check the features of the browser, that it's built against all required libs to support WebGL, html5 video and audio etc.

[favogt] Does it need to have SSL?
The best solution, imho, is to have a local server serving the pages. Depending on what are the requirements for the static page, apache might be overkill...
Is

while true; do echo -e "HTTP/1.1 200 OK\n\n Hello $(whoami)! Now is $(date)" | nc -l localhost 8080; done

enough or is it required a more complex case?

[dasantiago] Yes, it needs to check that the entire cert chain is valid. That is why I suggested https://www.opensuse.org. That rules out starting a local server though, as testing self-signed certs misses the point...

dasantiago:
The website code is on github. I will follow this with infra team, so we can have it hosted.
After talking to the infra team they suggested two hosts:

conncheck.opensuse.org
static.opensuse.org

We can set up our html5 page there.

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4250

All merged and SLE needles created. I checked many maintenance jobs, could be some will still fail because they have been triggered before the needles have been created. But it can work in general, see staging job: https://openqa.suse.de/tests/1411129#step/firefox/9