Project

General

Profile

action #80520

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

coordination #43706: [epic] Generate "download&use" docker image of openQA for SUSE QA

Automatic tests for our openQA containers - webUI only

Added by okurz 8 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-11-27
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

Without automatic tests we would not call any contributions properly long-term supportable. These tests could be very simple, e.g. something like podman run --rm -it .... openqa-worker --help or something.

Acceptance criteria

  • AC1: The webUI server is started completely without errors from container within CI tests, i.e. can render a common page e.g. /tests

Suggestions

  • Find out if OBS can execute tests for container images
  • Consider kiwi vs Dockerfile
  • Setup tests as part of the openQA CI (e.g. CircleCI or GitHub Actions)

Related issues

Copied to openQA Project - action #80682: Automatic tests for our openQA containers - worker onlyResolved2020-11-27

History

#1 Updated by cdywan 8 months ago

  • Description updated (diff)

#2 Updated by ilausuch 8 months ago

  • Status changed from New to In Progress
  • Assignee set to ilausuch

#3 Updated by ilausuch 8 months ago

In order to fix several things, this PR is necessary to run the web UI without the load balancer
https://github.com/os-autoinst/openQA/pull/3607

#4 Updated by okurz 8 months ago

  • Description updated (diff)

#5 Updated by okurz 8 months ago

  • Subject changed from Automatic tests for our openQA containers to Automatic tests for our openQA containers - webUI only
  • Description updated (diff)
  • Status changed from In Progress to Workable
  • Assignee deleted (ilausuch)

#6 Updated by okurz 8 months ago

  • Copied to action #80682: Automatic tests for our openQA containers - worker only added

#7 Updated by okurz 8 months ago

  • Target version changed from Ready to future

#8 Updated by ilausuch 8 months ago

This PR fix some problems with the SSL apache configuration when we run webUI container
https://github.com/os-autoinst/openQA/pull/3612

#9 Updated by cdywan 8 months ago

ilausuch wrote:

This PR fix some problems with the SSL apache configuration when we run webUI container
https://github.com/os-autoinst/openQA/pull/3612

Where by fixing problems you mean dropping the SSL configuration for Apache, right? That's what the PR proposes.

#10 Updated by ilausuch 8 months ago

  • Status changed from Workable to In Progress
  • Assignee set to ilausuch

#11 Updated by ilausuch 8 months ago

I created this new PR following some comments on the previous PR (that is replaced by this new)
https://github.com/os-autoinst/openQA/pull/3632
In this new PR I created different dockerfiles to cover different current scenarios:

  • container with all the services executed inside with apache as a reverse proxy + SSL. User has to provide the certificates
  • container with all the services executed inside with apache as a reverse proxy without SSL
  • container without apache and the services individually-executed controlled by the MODE environment variable. To be used within docker-compose or kubernetes

#12 Updated by ilausuch 8 months ago

for this test we will check that the web UI tries to connect to the DB

docker run -ti -v $(pwd)/t/data/database.ini:/data/conf/database.ini openqa_webui

With the result:

DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','',...) failed: could not connect to server: No such file or directory

Or we could try to provide a DB to the container and test this more properly.

#13 Updated by openqa_review 8 months ago

  • Due date set to 2020-12-24

Setting due date based on mean cycle time of SUSE QE Tools

#14 Updated by ilausuch 8 months ago

After a discussion today in a daily I will focus on create a test to cover the container without apache. My plan for the test is:

  • Build the web UI image
  • Create a DB container (postgresql)
  • Test that gru can start and listen on its port
  • Test that websockets can start and listen on its port
  • Test that livehandle can start and listen on its port
  • Test that webUI can start and listen on its port

This PR https://github.com/os-autoinst/openQA/pull/3632 will be postponed for the future if necessary

#15 Updated by ilausuch 8 months ago

To cover the plan of my last commentary I provide this PR https://github.com/os-autoinst/openQA/pull/3633

#16 Updated by ilausuch 8 months ago

I was exploring if we could use openQA jobs to create tests of our containers. Please see this job https://openqa.suse.de/tests/5173044
I think could be interesting because we could launch regular tests of our containers from VM created by openQA jobs. Yesterday I played with that and I achieved to run a container inside a job and also with privileged mode, so there are no barriers for that.
There are some considerations to take in consideration, and please, correct me if I am wrong: openQA is not integrated in the CI of the github so, if we want to test last changes on the dockerfiles in the last PR won't be useful, only could be use to regular testing.

On the other hand, we should consider two kind of testing:

  • Unit tests: For instance, that the web UI starts and is listening
  • Integration tests: we can launch a job, and all the parts work together til the end of the job.

#17 Updated by ilausuch 8 months ago

Seems that circleCI doesn't allows to run containers in privilege mode
https://circleci.com/docs/2.0/executor-types/#using-docker

And there is not configuration options to do that
https://circleci.com/docs/2.0/configuration-reference/#docker--machine--macos--windows-executor

#18 Updated by ilausuch 8 months ago

I am thinking in these solutions

Inside the github CI workflow we create a new step/action than launch a script that launches a job in openqa.suse.de / openqa.opensuse.org.
This script passes to this tests what is the branch we want to test as a parameter will be get by the perl test. The original script will remain waiting until the job has finished with a success, softfail or a failure reporting the result as result to the CI
The perl test clones the openQA repository with the specified branch, builds the container image, create the containers, and runs the tests of connection, etc.

There are other alternatives with kubernetes to this one, but I would like to explore this one.

#19 Updated by okurz 7 months ago

We discussed different options in today's daily meeting which can all in principle be followed independently:

  1. Same as in os-autoinst run static style checks on Dockerfiles within PR checks and run build+run steps when running on master, compare to https://github.com/os-autoinst/os-autoinst/pull/1602
  2. Extend the openQA-in-openQA tests in https://openqa.opensuse.org/group_overview/24 with a build+run test of our container definitions, e.g. boot a Tumbleweed system, build container images with podman and run them
  3. Similar as the existing openQA-in-openQA tests create a new trigger script which looks for new container image files defined in http://download.opensuse.org/repositories/devel:/openQA/containers15.2/ and trigger according tests in openQA . For this I suggest to not trigger the same as we do in Jenkins but instead use github actions

#20 Updated by ilausuch 7 months ago

Trying to run the test this in OpenQA. I had always the same issue with the timeouts with the command docker run and podman run.
The problem of these VMs are that we don't have hardware virtualization, so this is really very slow

QEMU: Checking for hardware virtualization : FAIL (Only emulated CPUs are available, performance will be significantly limited)

With @Geor helping we are exploring baremetal machines to do this test.

#22 Updated by ilausuch 6 months ago

Initial tests using the openQA repo instead of opensuse
PR: https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/60
Test: http://10.161.229.197/tests/184

Now I am using docker because I have to solve some problems with the podman version resolving the name of the DB

#23 Updated by ilausuch 6 months ago

I change the status of the PR to be reviewed.

#24 Updated by cdywan 6 months ago

  • Due date changed from 2020-12-24 to 2021-01-22

#26 Updated by cdywan 6 months ago

  • Status changed from In Progress to Feedback

#27 Updated by ilausuch 6 months ago

This test is regular tested in openqa.opensuse.org
https://openqa.opensuse.org/tests/1607610

#28 Updated by ilausuch 6 months ago

  • Status changed from Feedback to Resolved

#29 Updated by okurz 6 months ago

  • Status changed from Resolved to Feedback

The openQA-in-openQA tests are good but I thought we talked about this often enough and decided that we also want at least a subset of tests within the CI tests before merge, see #80520#note-19

#30 Updated by ilausuch 6 months ago

Yes, you're right

#31 Updated by cdywan 6 months ago

My understanding was, CI didn't work so well, and containers do the job. Otherwise we wouldn't have gone for openQA tests at this point.

#32 Updated by okurz 6 months ago

cdywan wrote:

"CI didn't work so well, and containers do the job."

Hm, not sure what you mean by that. Maybe you can clarify what you mean. How I see the current situation is like this: With openQA-in-openQA tests we can do real functional tests which is good. However, we can still have more feedback before merge, part of the pull request testing CI tests. In #80520#note-19 I mentioned the options and referenced the os-autoinst pull request that tests container definitions as part of such CI tests. In one of the followup meetings we clarified (I think) that we should also have such tests for openQA. They will not do the same as the openQA-in-openQA tests but check that the Dockerfiles as versioned within our git repo are usable, i.e. buildable.

#33 Updated by cdywan 6 months ago

okurz wrote:

cdywan wrote:

"CI didn't work so well, and containers do the job."

Hm, not sure what you mean by that. Maybe you can clarify what you mean.

There were problems with the proposed tests on CI, with building and running containers, both with podman and docker, concerning basics like permissions and what user to run as.

How I see the current situation is like this: With openQA-in-openQA tests we can do real functional tests which is good. However, we can still have more feedback before merge, part of the pull request testing CI tests. In #80520#note-19 I mentioned the options and referenced the os-autoinst pull request that tests container definitions as part of such CI tests. In one of the followup meetings we clarified (I think) that we should also have such tests for openQA. They will not do the same as the openQA-in-openQA tests but check that the Dockerfiles as versioned within our git repo are usable, i.e. buildable.

If you're indeed talking about static validation of container files I consider that out of scope for this ticket. The webUI server is started completely without errors is about running the container successfully.

#34 Updated by okurz 6 months ago

cdywan wrote:

okurz wrote:

cdywan wrote:

"CI didn't work so well, and containers do the job."

Hm, not sure what you mean by that. Maybe you can clarify what you mean.

There were problems with the proposed tests on CI, with building and running containers, both with podman and docker, concerning basics like permissions and what user to run as.

How I see the current situation is like this: With openQA-in-openQA tests we can do real functional tests which is good. However, we can still have more feedback before merge, part of the pull request testing CI tests. In #80520#note-19 I mentioned the options and referenced the os-autoinst pull request that tests container definitions as part of such CI tests. In one of the followup meetings we clarified (I think) that we should also have such tests for openQA. They will not do the same as the openQA-in-openQA tests but check that the Dockerfiles as versioned within our git repo are usable, i.e. buildable.

If you're indeed talking about static validation of container files I consider that out of scope for this ticket. The webUI server is started completely without errors is about running the container successfully.

Well, the "static validation" is merely an implementation consequence when we can not have full functional tests within the PR CI tests.

I think the addition to the openQA-in-openQA tests are great. If you want to split out the mentioned idea into another ticket and call this one done, sure, we can do that. I would just prefer to have some tests for Dockerfiles being executed as part of every PR test.

#35 Updated by cdywan 6 months ago

okurz wrote:

I think the addition to the openQA-in-openQA tests are great. If you want to split out the mentioned idea into another ticket and call this one done, sure, we can do that. I would just prefer to have some tests for Dockerfiles being executed as part of every PR test.

Fully agree. Between the two of us, I really want to see those static checks, too. I'm just trying to keep ACs in focussed and workable chunks.

I filed #88451

#36 Updated by ilausuch 6 months ago

  • Status changed from Feedback to Resolved

Because of the AC for this ticket is covered and Oliver's suggestions are in the AC of the new ticket, I change the status of this one to resolve.

#37 Updated by okurz 4 months ago

  • Due date deleted (2021-01-22)

Also available in: Atom PDF