Project

General

Profile

Actions

action #20228

closed

[qam]stop relying on html5test.com

Added by pgeorgiadis about 7 years ago. Updated over 6 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Target version:
-
Start date:
2017-07-03
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-12-SP1-Server-DVD-Updates-x86_64-qam-gnome@64bit fails in
firefox

Relying on external webservers is not optimal. Even visiting the same webpage (in that case html5test.com) has been proven to be quite unstable, since within 24h timeframe, their server triggers different behavior of Firefox' Tracking Protection mechanism. As a result, yesterday, the same test passed, but today is failing. Also, re-adjusting the needle is not a option, since by adding the shield icon, the URL gets shifted to the more right direction (thus the needle with missmatch).

As a solution, I would propose to replace 'html5test.com' with a local webpage generated by a webserver. Some approaches:

zypper in -t pattern lamp_server (this will also install 'apache2-example-pages' pkg which will automatically create an index.html accessible by localhost)
systemctl start apache2
curl -s localhost | grep "It works!"
// Now that you know that your page is accessible, fire Firefox and do your tests

Alternatively, we could build a docker image that provides a pre-configured website by default:
zypper in docker
docker run

Another approach suggested by @asmorodskyi would be to use https://www.lighttpd.net/ . The benefit is that in this approach we will not rely on zypper to install apache2 but just rar.gz as test resource. In that case we are not adding additional reason to fail the test.

What do you think?

Reproducible

Fails since (at least) Build 20170703-1 (current job)

Expected result

Last good: 20170702-1 (or more recent)

Further details

Always latest result in this scenario: latest


Related issues 1 (0 open1 closed)

Is duplicate of openQA Tests - action #14848: [qam][sle][opensuse][functional][x11regressions]test fails in firefox with what looks like temporary problem to access html5test.comResolvedokurz2016-11-16

Actions
Actions #1

Updated by pgeorgiadis about 7 years ago

  • Description updated (diff)
Actions #2

Updated by pgeorgiadis about 7 years ago

  • Description updated (diff)
Actions #3

Updated by SLindoMansilla about 7 years ago

  • Category changed from Bugs in existing tests to Enhancement to existing tests

Good point it would be a good enhancement to the test. But it is not a bug in the test.

Actions #4

Updated by okurz about 7 years ago

  • Subject changed from stop relying on html5test.com to [qam]stop relying on html5test.com
Actions #5

Updated by pluskalm about 7 years ago

  • Priority changed from Normal to Low
Actions #6

Updated by pluskalm almost 7 years ago

  • Assignee set to dasantiago
Actions #7

Updated by favogt almost 7 years ago

  • Priority changed from Low to Normal

Relying on https://html5test.com proved to be pretty bad once again, as currently every single firefox/chromium test fails because of an expires SSL certificate.
I'll make a PR shortly that uses http in those tests, but that will require new needles due to the URL change (and changed security)...
For a long-term solution I'd suggest adding a static html page somewhere on https://www.opensuse.org, which should be reachable from everywhere and be reliable enough, at least we can fix it ourselves if it breaks.

Actions #8

Updated by dasantiago almost 7 years ago

Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?

Does it need to have SSL?

The best solution, imho, is to have a local server serving the pages. Depending on what are the requirements for the static page, apache might be overkill...

Is

while true; do echo -e "HTTP/1.1 200 OK\n\n Hello $(whoami)! Now is $(date)" | nc -l localhost 8080; done

enough or is it required a more complex case?

Actions #9

Updated by favogt almost 7 years ago

Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?

IMO it should check the features of the browser, that it's built against all required libs to support WebGL, html5 video and audio etc.

Does it need to have SSL?

Yes, it needs to check that the entire cert chain is valid. That is why I suggested https://www.opensuse.org.
That rules out starting a local server though, as testing self-signed certs misses the point...

Currently html5test.com is down again as api.whichbrowser.net (which had the expired cert) now completely refuses to respond and all openqa tests will once again fail...

Actions #10

Updated by dasantiago almost 7 years ago

favogt wrote:

Does the static page needs to contain any specific content? Can it be just a simple "Hello world" ? Or does it need to have images,css, js... ?

IMO it should check the features of the browser, that it's built against all required libs to support WebGL, html5 video and audio etc.

Does it need to have SSL?

Yes, it needs to check that the entire cert chain is valid. That is why I suggested https://www.opensuse.org.
That rules out starting a local server though, as testing self-signed certs misses the point...

Currently html5test.com is down again as api.whichbrowser.net (which had the expired cert) now completely refuses to respond and all openqa tests will once again fail...

The website code is on github. I will follow this with infra team, so we can have it hosted.

Actions #11

Updated by favogt almost 7 years ago

Note that not html5test.com itself was down, but some of the third-party APIs it uses.
In this case, api.whichbrowser.net (also on GitHub).

Actions #12

Updated by dasantiago almost 7 years ago

After talking to the infra team they suggested two hosts:

  1. conncheck.opensuse.org
  2. static.opensuse.org

We can set up our html5 page there.

Actions #13

Updated by dasantiago almost 7 years ago

unfortunately, both html5test is not a static page :-(

https://github.com/WebPlatformTest/HTML5test/blob/version-8.0/backend/index.php#L60

Actions #14

Updated by vsvecova over 6 years ago

Same issue has been happening again for the past ~3 days.
Examples:

https://openqa.suse.de/tests/1351200#step/firefox/10
https://openqa.suse.de/tests/1351325#step/firefox/10
https://openqa.suse.de/tests/1351146#step/firefox/10

Looks like the problem has been coming back periodically over the past year or so; cf. closed older ticket poo#12282.

Btw. there is also another recent duplicate ticket poo#26976.

Actions #15

Updated by okurz over 6 years ago

  • Status changed from New to Rejected

incorporated all content into duplicatee #14848

Actions #16

Updated by okurz over 6 years ago

  • Is duplicate of action #14848: [qam][sle][opensuse][functional][x11regressions]test fails in firefox with what looks like temporary problem to access html5test.com added
Actions

Also available in: Atom PDF