QA » openQA Project

Today we discussed several things needed in order to have proper tests.

1. Fixtures: that is, example data in both the database and the testresults directory, leaving the production database and testresults untouched, of course.
2. Setup and teardown mechanisms: that is, tests needs to ensure the dataset is "clean" before running and after finishing its execution.
3. Some tests have a time-dependency on the fixtures.

Some useful stuff:

• https://github.com/tempire/mojoexample A quite good example of a Mojo app using fixtures and an in-memory sqlite3 database for running the tests. Let's steal some code!
• http://search.cpan.org/~jjnapiork/Test-DBIx-Class-0.41/lib/Test/DBIx/Class.pm Offers mechanisms to create temporary databases, load fixtures and fake results (useful for time dependent tests)
• http://search.cpan.org/~rjbs/Test-Class-0.41/lib/Test/Class.pm Setup and teardown for Test:More
• https://metacpan.org/pod/Test::SetupTeardown Another alternative to the former

Playing with https://github.com/tempire/mojoexample and having a look at the implementation of its tests suite. It's exactly what I had in mind for ours, so we can grab some ideas and code.

I will describe here the ideas I want to copy/adapt from https://github.com/tempire/MojoExample

• First of all, they have a facility Test::Database package providing the following functionality:

   # Creates an sqlite3 test.db database from DBIC Schema AND load the fixtures
   my $schema = Test::Database->new->create(Schema => 'test.db');
   # Creates an in-memory sqlite3 database from DBIC Schema AND load the fixtures
   my $schema = Test::Database->new->create(Schema => ':memory:');

• They define a attribute for the application called 'schema' and a helper called 'db' for accessing it from the controllers. So they can:

  # At any point
  $theApp->schema # The first time it's called without having been explicitly assigned, it will default to the production db

  # In a controller
  $self->db->resultset('Photoset')->search

  # At the beginning of every test that is independent of Mojo (model tests)
  my $schema = Test::Database->new->create(Schema => ':memory:')
  $schema->test_whatever

  # At the beginning of Mojo tests
  my $schema = Test::Database->new->create(Schema => 'test.db');
  my $t = Test::Mojo->new('NameOfTheApp');
  $t->app->schema($schema);

In this way, they have a single point to access the database in the application, an easy way to create a temporary database with all the test fixtures (that are stored in 't/fixtures') loaded and a convenient way to change the database in use during execution time (handy for the tests).

Fixtures and volatile test databases. Almost everything we need.

Old tests fixed to use the new temporary database mechanism. 'make test' is not needed anymore, the good old 'script/openqa test' manages everything now.

Added support for tests in subdirectories. Took me a while. I had a problem, I tried to solve it with Perl and regular expressions... you know the rest.

Test::Mojo works nicely 'emulating' a web browser as long as there is no Javascript involved. The problem is that half of the openQA UI relies on Javascript. We have two options for testing the javascript

1. Keep the current approach for testing server responses and a different test suite just for testing the Javascript using Mocha, Jasmine or something similar.
2. Add javascript support for our perl-based test suite so we can simulate a more full-featured browser.

I have been researching about the second idea (that is, looking for a perl equivalent to Ruby's Capybara) and apparently we have three main options (apart from more custom-made solutions):

• Test::WWW:Mechanize::Mojo, a library that merges Test::Mojo and Test::WWW::Mechanize.
• Brownie, a browser integration framework which supports two drivers (backends to really drive the interaction with the web server): WWW::Mechanize and Selenium RemoteWebDriver
• Wight, a browser integration framework with PhantomJS as backend.

PhantomJS is awesome and we already have packages in OBS. It would be my first option. Unfortunately Wight looks way less mature than Brownie.

Mechanize does not support javascript, unless we can use WWW::Scripter::Plugin::Javascript, Gtk2::WebKit::Mechanize or WWW-Mechanize-Firefox. All of them are supposed to be implementations of Mechanize. If they really work as drop-in replacements, Test::WWW:Mechanize::Mojo and Brownie+Mechanize would become nice options for our purposes.

Brownie+Selenium RemoteWebDriver would mean needing a Selenium server to run the tests (a small Java monster). Or not? The good news are that PhantomJS implements the WebDriver Wire protocol, which means that can act as a Selenium server replacement, so in theory you could use Selenium::Remote::Driver with PhantomJS, with no real Selenium at all. On the other hand, we could also try WWW::WebKit which is supposed to be a drop-in replacement for WWW::Selenium. If it works, we could use Brownie+RemoteWebDriver without needing a server running at all, none Selenium or PhantomJS.

In my opinion, we should really try to use Test::WWW:Mechanize::Mojo with a javascript enabled Mechanize replacement as a first option.

Fixing a bug in massive job restarting and adding tests for API:V1:Jobs in the process https://github.com/os-autoinst/openQA/pull/87

https://github.com/os-autoinst/openQA/pull/177 is a PoC for using phantomjs together with Selenium::Remote::Driver

isos tests ("assets") considered done with
commit 981dde70fda4e27e0850d5091be32e76b777a668
Author: Ondrej Holecek oholecek@suse.com
Date: Thu Apr 23 10:47:07 2015 +0200

add asset tests

- check proper asset assignements
- check job is cloned if asset creator is cloned