action #1272

Write some test cases for webUI

Added by ancorgs about 6 years ago. Updated over 1 year ago.

Status:ResolvedStart date:27/01/2014
Priority:NormalDue date:
Assignee:-% Done:

60%

Category:Feature requestsEstimated time:30.00 hours
Target version:QA - future
Difficulty:
Duration:

Description

This should probably not be a separate task, since should be part of #1271. In any case, after the initial porting some unit tests and integration tests (I'm using Rails terminology here, that can be different to the Mojo one) will be needed in order to be able to verify that the port works as expected and to be able to add new features.

That means learning to write tests in Mojolicious, of course.


Checklist

  • Interface test: results of a finished test
  • Interface test: live view (running test)
  • Interface test: list of tests
  • Interface test: buildview
  • Responses from running controller: modlist, status, livelog and streaming
  • Uploadlog
  • Interface test: needle editor
  • Interface test: audioview
  • Interface test: needle diff
  • Interface test: view test source code
  • Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y
  • API test: jobs
  • API tests: workers
  • API tests: isos

Related issues

Duplicated by openQA Project - action #1715: write api openqa tests Rejected 25/02/2014

History

#1 Updated by alarrosa about 6 years ago

  • Target version set to Sprint 01

#2 Updated by alarrosa about 6 years ago

  • Assignee set to cwh

#3 Updated by alarrosa about 6 years ago

  • Target version changed from Sprint 01 to Sprint 02

#4 Updated by cwh about 6 years ago

  • % Done changed from 0 to 10

#5 Updated by ancorgs about 6 years ago

Today we discussed several things needed in order to have proper tests.

  1. Fixtures: that is, example data in both the database and the testresults directory, leaving the production database and testresults untouched, of course.
  2. Setup and teardown mechanisms: that is, tests needs to ensure the dataset is "clean" before running and after finishing its execution.
  3. Some tests have a time-dependency on the fixtures.

Some useful stuff:

#6 Updated by ancorgs about 6 years ago

Playing with https://github.com/tempire/mojoexample and having a look at the implementation of its tests suite. It's exactly what I had in mind for ours, so we can grab some ideas and code.

#7 Updated by ancorgs about 6 years ago

I will describe here the ideas I want to copy/adapt from https://github.com/tempire/MojoExample

  • First of all, they have a facility Test::Database package providing the following functionality:
   # Creates an sqlite3 test.db database from DBIC Schema AND load the fixtures
   my $schema = Test::Database->new->create(Schema => 'test.db');
   # Creates an in-memory sqlite3 database from DBIC Schema AND load the fixtures
   my $schema = Test::Database->new->create(Schema => ':memory:');

  • They define a attribute for the application called 'schema' and a helper called 'db' for accessing it from the controllers. So they can:
  # At any point
  $theApp->schema # The first time it's called without having been explicitly assigned, it will default to the production db

  # In a controller
  $self->db->resultset('Photoset')->search

  # At the beginning of every test that is independent of Mojo (model tests)
  my $schema = Test::Database->new->create(Schema => ':memory:')
  $schema->test_whatever

  # At the beginning of Mojo tests
  my $schema = Test::Database->new->create(Schema => 'test.db');
  my $t = Test::Mojo->new('NameOfTheApp');
  $t->app->schema($schema);

In this way, they have a single point to access the database in the application, an easy way to create a temporary database with all the test fixtures (that are stored in 't/fixtures') loaded and a convenient way to change the database in use during execution time (handy for the tests).

Fixtures and volatile test databases. Almost everything we need.

#8 Updated by ancorgs about 6 years ago

  • Status changed from New to In Progress
  • % Done changed from 10 to 20

After some pair programming (Christopher and myself), we already have a mechanism to run tests in an automatically created test database with some fixtures loaded from files. Let's write tons of tests.

#9 Updated by ancorgs about 6 years ago

Old tests fixed to use the new temporary database mechanism. 'make test' is not needed anymore, the good old 'script/openqa test' manages everything now.

#10 Updated by ancorgs about 6 years ago

  • % Done changed from 20 to 30

Test framework improved to also use temporary example testresults. Listing test adapted to use the new example data. Still not committed because I'm improving the example testresults.

#11 Updated by ancorgs about 6 years ago

  • % Done changed from 30 to 50

Now we have a test for '/tests' which use example fixtures and example testresults (including succeeded and failed tests) in a safe way with time-dependent information (to test the time filter). Will all that in place, writing more tests should be almost peanuts (except those needing Javascript).

#12 Updated by ancorgs about 6 years ago

  • Checklist set to [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [ ] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [ ] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [ ] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [ ] API test: jobs, [ ] API tests: workers, [ ] API tests: isos

I created a list of tests we should implement based on the already existing infrastructure

#13 Updated by ancorgs about 6 years ago

Added support for tests in subdirectories. Took me a while. I had a problem, I tried to solve it with Perl and regular expressions... you know the rest.

#14 Updated by alarrosa almost 6 years ago

  • Target version changed from Sprint 02 to Sprint 03

#15 Updated by ancorgs almost 6 years ago

Test::Mojo works nicely 'emulating' a web browser as long as there is no Javascript involved. The problem is that half of the openQA UI relies on Javascript. We have two options for testing the javascript

  1. Keep the current approach for testing server responses and a different test suite just for testing the Javascript using Mocha, Jasmine or something similar.
  2. Add javascript support for our perl-based test suite so we can simulate a more full-featured browser.

I have been researching about the second idea (that is, looking for a perl equivalent to Ruby's Capybara) and apparently we have three main options (apart from more custom-made solutions):

  • Test::WWW:Mechanize::Mojo, a library that merges Test::Mojo and Test::WWW::Mechanize.
  • Brownie, a browser integration framework which supports two drivers (backends to really drive the interaction with the web server): WWW::Mechanize and Selenium RemoteWebDriver
  • Wight, a browser integration framework with PhantomJS as backend.

PhantomJS is awesome and we already have packages in OBS. It would be my first option. Unfortunately Wight looks way less mature than Brownie.

Mechanize does not support javascript, unless we can use WWW::Scripter::Plugin::Javascript, Gtk2::WebKit::Mechanize or WWW-Mechanize-Firefox. All of them are supposed to be implementations of Mechanize. If they really work as drop-in replacements, Test::WWW:Mechanize::Mojo and Brownie+Mechanize would become nice options for our purposes.

Brownie+Selenium RemoteWebDriver would mean needing a Selenium server to run the tests (a small Java monster). Or not? The good news are that PhantomJS implements the WebDriver Wire protocol, which means that can act as a Selenium server replacement, so in theory you could use Selenium::Remote::Driver with PhantomJS, with no real Selenium at all. On the other hand, we could also try WWW::WebKit which is supposed to be a drop-in replacement for WWW::Selenium. If it works, we could use Brownie+RemoteWebDriver without needing a server running at all, none Selenium or PhantomJS.

In my opinion, we should really try to use Test::WWW:Mechanize::Mojo with a javascript enabled Mechanize replacement as a first option.

#16 Updated by ancorgs almost 6 years ago

  • Checklist changed from [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [ ] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [ ] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [ ] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [ ] API test: jobs, [ ] API tests: workers, [ ] API tests: isos to [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [ ] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [ ] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [ ] API test: jobs, [ ] API tests: workers, [ ] API tests: isos

Tests for file controller added. Two bugs killed in the process.

#17 Updated by alarrosa almost 6 years ago

  • Target version changed from Sprint 03 to Sprint 04

#18 Updated by alarrosa almost 6 years ago

  • Target version changed from Sprint 04 to Sprint 05

#19 Updated by alarrosa almost 6 years ago

  • Target version changed from Sprint 05 to Sprint 06

#20 Updated by alarrosa almost 6 years ago

  • Target version deleted (Sprint 06)

#21 Updated by ancorgs almost 6 years ago

  • Checklist changed from [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [ ] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [ ] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [ ] API test: jobs, [ ] API tests: workers, [ ] API tests: isos to [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [ ] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [ ] API test: jobs, [ ] API tests: workers, [ ] API tests: isos

Tests for the new buildview included in #1385. One strike more in the checklist.

#22 Updated by ancorgs over 5 years ago

Fixing a bug in massive job restarting and adding tests for API:V1:Jobs in the process https://github.com/os-autoinst/openQA/pull/87

#23 Updated by coolo over 5 years ago

  • Status changed from In Progress to New
  • Assignee deleted (cwh)
  • Priority changed from Immediate to Normal

#24 Updated by coolo over 5 years ago

  • Category set to 124

#25 Updated by coolo about 5 years ago

https://github.com/os-autoinst/openQA/pull/177 is a PoC for using phantomjs together with Selenium::Remote::Driver

#26 Updated by okurz about 4 years ago

  • Checklist changed from [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [ ] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [ ] API test: jobs, [ ] API tests: workers, [ ] API tests: isos to [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [x] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [ ] API test: jobs, [ ] API tests: workers, [ ] API tests: isos

needle editor test fixed with

commit d3a6e159bc39da8ac9740d49961080dac7f3e9a9
Author: Max Lin mlin@suse.com
Date: Wed Mar 18 17:53:25 2015 +0800

Improve needle editor test coverage, action#6562

Testing needle editor based on Selenium::Remote:Driver and phantomjs.

#27 Updated by okurz about 4 years ago

  • Checklist set to [x] API tests: isos

#28 Updated by okurz about 4 years ago

isos tests ("assets") considered done with
commit 981dde70fda4e27e0850d5091be32e76b777a668
Author: Ondrej Holecek oholecek@suse.com
Date: Thu Apr 23 10:47:07 2015 +0200

add asset tests

- check proper asset assignements
- check job is cloned if asset creator is cloned

#29 Updated by okurz about 4 years ago

  • Assignee set to dheidler

@dheidler as you currently have only one task assigned you might want to keep this as a "fallback" task :-) Please could you check on the current status of tests and update this ticket

#30 Updated by dheidler about 4 years ago

  • Checklist set to [x] API tests: workers

#31 Updated by dheidler about 4 years ago

  • Checklist set to [ ] API tests: workers

#32 Updated by dheidler about 4 years ago

  • Checklist changed from [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [x] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [ ] API test: jobs, [ ] API tests: workers, [x] API tests: isos to [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [x] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [x] API test: jobs, [x] API tests: workers, [x] API tests: isos
  • % Done changed from 50 to 60
./t/api/01-workers.t ...................... ok
...
./t/api/04-jobs.t ......................... ok

#33 Updated by dheidler about 4 years ago

  • Checklist changed from [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [x] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [ ] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [x] API test: jobs, [x] API tests: workers, [x] API tests: isos to [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [x] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [x] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [x] API test: jobs, [x] API tests: workers, [x] API tests: isos

t/ui/03-source.t

#34 Updated by dheidler over 3 years ago

  • Assignee changed from dheidler to okurz

#35 Updated by okurz over 3 years ago

  • Checklist changed from [ ] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [x] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [x] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [x] API test: jobs, [x] API tests: workers, [x] API tests: isos to [x] Interface test: results of a finished test, [ ] Interface test: live view (running test), [x] Interface test: list of tests, [x] Interface test: buildview, [ ] Responses from running controller: modlist, status, livelog and streaming, [ ] Uploadlog, [x] Interface test: needle editor, [ ] Interface test: audioview, [ ] Interface test: needle diff, [x] Interface test: view test source code, [x] Responses from file controller: images/x, file/x, logfile/x, diskimages/x, needles/x/y, [x] API test: jobs, [x] API tests: workers, [x] API tests: isos
  • Assignee deleted (okurz)

See t/ui/18-tests-details.t since

commit 7fac90a
Author: Oliver Kurz okurz@suse.de
Date: Fri Jun 10 14:57:37 2016 +0200

Add proper test for wait_serial output on web UI

New fake test results are added to one existing job resembling a bit more how
a real "sshfs" test looks like including a wait_serial output.
'tests-src' is renamed to 'tests-details', testing both src window as well as
wait_serial output. It can be extended to test the custom test fails which
also have sparingly, e.g. when the backend dies.

#36 Updated by mkittler over 3 years ago

  • Checklist set to [x] Interface test: needle diff

#37 Updated by mkittler over 3 years ago

  • Checklist set to [ ] Interface test: needle diff

#38 Updated by okurz about 3 years ago

  • Target version set to future

#39 Updated by EDiGiacinto almost 3 years ago

  • Checklist set to [x] Interface test: live view (running test)

#40 Updated by EDiGiacinto almost 3 years ago

  • Checklist set to [ ] Interface test: live view (running test)

#41 Updated by coolo over 2 years ago

  • Status changed from New to Resolved

Closing this epic - we are pretty good already with the full tests and with new stuff added.

#42 Updated by okurz over 1 year ago

  • Target version changed from future to future

Also available in: Atom PDF