QA » openQA Project » openQA Tests

Primitive notes on point #6 of the description:

Selenium itself cannot recreate, say, the current firefox_html5 test as it it now. One reason is that this test actually uses elements outside of the page DOM (like the tab bar), which selenium cannot access. Another is that the current test is heavily relying on image recognition, which again selenium cannot test. Both the above functions can be achieved with other, 3rd party tools, but I consider this out of scope.

The question is - is the goal here a mix between selenium and openqa? Is a setup where selenium controls the browser and openqa verifies needles what is wanted? Or should we rewrite the test / create a new test case suitable for selenium?

Perhaps we would benefit from a more specific test case. I will keep exploring in the meanwhile.

Bill, give it a look at the tomcat tests then. There are parts of the test where openQA is doing setup or changing elements behind the scenes, but the web component should be testable (It should be testable on both, tomcat and HTML5 test), the intention is to answer the questions stated on this ticket.

Could you document what you've done so far, and link PR's even if they aren't in a mergeable state, so that we could have a rough idea of how that could look like during the review?. (Note, this doesn't need to be done, done and closed by the review, as we'll most likely carry it over to the next sprint and close it then).

Settle down on what's needed for features

Suggested PoC in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14905

Conclusions + results, as asked in the ticket description:

1. There are multiple ways (and languages) to use for selenium testing. I went with setting up the selenium webdriver for python and writing the tests as a python script, as the most readily available and easy solution. Indeed, it is easy and quick to do so (except for some caveats, see further below)

2. I tried a couple of different setups for the actual testing. Specifically:
   a) Python script runs tests, uses pytest to assert various expected results
   b) Python script runs tests, writes results to a file, perl mudule parses results from file
   c) Python script runs tests, writes results to a file,perl module only checks last line of the file (OK/FAIL) and points to it if tests failed
   From the above:
   I settled down for c, because of the following reasons:
   (a) just felt too much of a hack, since pytest is developed with software/function testing in mind, and not parsing results of tests. It can be done - but it requires even more python module installs, it may face problems with python versions (which may or may not be available in a given SUT), it needs tweaks to work in the intended for our testing way, and ultimately doesn't offer anything more than a different way to deliver the same results.
   (b) was doing a lot of specialized parsing in the perl module, which enforced a lot of rules on anyone writing a new selenium test on how results are displayed, and reusability was compromised

3. Once the above has been decided, writing tests is easy. Selenium provides a very user-friendly way of navigating html/js pages. Depending on what the user is testing, additional downloads of drivers may be required (e.g. geckodriver for firefox). More further below.

4. Yes, I think that if something can be tested with selenium, tests of arbitrary length and scope can be implemented successfully.

5. The architecture I implemented is the one vaguely described above:
   (-) .pm module installs requirements (e.g. puthon3, selenium webdriver, posible additional stuff), grabs the specific python script for the test in question, and writes it's output to a file
   (-) python script runs all the actual testing, outputting results on the go. It must be ensured that even if the script encounters errors, it won't die, but instead at least output a last line containing the word "FAIL" before it ends.
   (-) the perl module uploads results file in openqa, and checks last line to determine if tests were a success or failure

6. (-) pros:Testing with selenium comes with all the positives that stem out of using a dedicated and widely used framework for web testing. Writing tests is easy, testing is fast, and selenium provides a number of ways to deal with common problems (late element loading, js messing with automation, etc).
   (-) cons: it may require additional maintenance in some cases. Specifically for firefox, the link to download the latest geckodriver must be used inside the perl module, which may need to be updated from time to time.

7. I believe that using selenium for web testing provides more solutions than problems. The ease and speed of running tests, especially after an architecture and basic rules are established, together with the features a dedicated web framework provides, and the readily available information on using selenium on the web, makes it a worthy addition.