action #80520
closedcoordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
coordination #43706: [epic] Generate "download&use" docker image of openQA for SUSE QA
Automatic tests for our openQA containers - webUI only
0%
Description
Motivation¶
Without automatic tests we would not call any contributions properly long-term supportable. These tests could be very simple, e.g. something like podman run --rm -it .... openqa-worker --help
or something.
Acceptance criteria¶
- AC1: The webUI server is started completely without errors from container within CI tests, i.e. can render a common page e.g. /tests
Suggestions¶
- Find out if OBS can execute tests for container images
- Consider kiwi vs Dockerfile
- Setup tests as part of the openQA CI (e.g. CircleCI or GitHub Actions)
Updated by ilausuch almost 4 years ago
- Status changed from New to In Progress
- Assignee set to ilausuch
Updated by ilausuch almost 4 years ago
In order to fix several things, this PR is necessary to run the web UI without the load balancer
https://github.com/os-autoinst/openQA/pull/3607
Updated by okurz almost 4 years ago
- Subject changed from Automatic tests for our openQA containers to Automatic tests for our openQA containers - webUI only
- Description updated (diff)
- Status changed from In Progress to Workable
- Assignee deleted (
ilausuch)
Updated by okurz almost 4 years ago
- Copied to action #80682: Automatic tests for our openQA containers - worker only added
Updated by ilausuch almost 4 years ago
This PR fix some problems with the SSL apache configuration when we run webUI container
https://github.com/os-autoinst/openQA/pull/3612
Updated by livdywan almost 4 years ago
ilausuch wrote:
This PR fix some problems with the SSL apache configuration when we run webUI container
https://github.com/os-autoinst/openQA/pull/3612
Where by fixing problems you mean dropping the SSL configuration for Apache, right? That's what the PR proposes.
Updated by ilausuch almost 4 years ago
- Status changed from Workable to In Progress
- Assignee set to ilausuch
Updated by ilausuch almost 4 years ago
I created this new PR following some comments on the previous PR (that is replaced by this new)
https://github.com/os-autoinst/openQA/pull/3632
In this new PR I created different dockerfiles to cover different current scenarios:
- container with all the services executed inside with apache as a reverse proxy + SSL. User has to provide the certificates
- container with all the services executed inside with apache as a reverse proxy without SSL
- container without apache and the services individually-executed controlled by the MODE environment variable. To be used within docker-compose or kubernetes
Updated by ilausuch almost 4 years ago
for this test we will check that the web UI tries to connect to the DB
docker run -ti -v $(pwd)/t/data/database.ini:/data/conf/database.ini openqa_webui
With the result:
DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','',...) failed: could not connect to server: No such file or directory
Or we could try to provide a DB to the container and test this more properly.
Updated by openqa_review almost 4 years ago
- Due date set to 2020-12-24
Setting due date based on mean cycle time of SUSE QE Tools
Updated by ilausuch almost 4 years ago
After a discussion today in a daily I will focus on create a test to cover the container without apache. My plan for the test is:
- Build the web UI image
- Create a DB container (postgresql)
- Test that gru can start and listen on its port
- Test that websockets can start and listen on its port
- Test that livehandle can start and listen on its port
- Test that webUI can start and listen on its port
This PR https://github.com/os-autoinst/openQA/pull/3632 will be postponed for the future if necessary
Updated by ilausuch almost 4 years ago
To cover the plan of my last commentary I provide this PR https://github.com/os-autoinst/openQA/pull/3633
Updated by ilausuch almost 4 years ago
I was exploring if we could use openQA jobs to create tests of our containers. Please see this job https://openqa.suse.de/tests/5173044
I think could be interesting because we could launch regular tests of our containers from VM created by openQA jobs. Yesterday I played with that and I achieved to run a container inside a job and also with privileged mode, so there are no barriers for that.
There are some considerations to take in consideration, and please, correct me if I am wrong: openQA is not integrated in the CI of the github so, if we want to test last changes on the dockerfiles in the last PR won't be useful, only could be use to regular testing.
On the other hand, we should consider two kind of testing:
- Unit tests: For instance, that the web UI starts and is listening
- Integration tests: we can launch a job, and all the parts work together til the end of the job.
Updated by ilausuch almost 4 years ago
Seems that circleCI doesn't allows to run containers in privilege mode
https://circleci.com/docs/2.0/executor-types/#using-docker
And there is not configuration options to do that
https://circleci.com/docs/2.0/configuration-reference/#docker--machine--macos--windows-executor
Updated by ilausuch almost 4 years ago
I am thinking in these solutions
Inside the github CI workflow we create a new step/action than launch a script that launches a job in openqa.suse.de / openqa.opensuse.org.
This script passes to this tests what is the branch we want to test as a parameter will be get by the perl test. The original script will remain waiting until the job has finished with a success, softfail or a failure reporting the result as result to the CI
The perl test clones the openQA repository with the specified branch, builds the container image, create the containers, and runs the tests of connection, etc.
There are other alternatives with kubernetes to this one, but I would like to explore this one.
Updated by okurz almost 4 years ago
We discussed different options in today's daily meeting which can all in principle be followed independently:
- Same as in os-autoinst run static style checks on Dockerfiles within PR checks and run build+run steps when running on master, compare to https://github.com/os-autoinst/os-autoinst/pull/1602
- Extend the openQA-in-openQA tests in https://openqa.opensuse.org/group_overview/24 with a build+run test of our container definitions, e.g. boot a Tumbleweed system, build container images with podman and run them
- Similar as the existing openQA-in-openQA tests create a new trigger script which looks for new container image files defined in http://download.opensuse.org/repositories/devel:/openQA/containers15.2/ and trigger according tests in openQA . For this I suggest to not trigger the same as we do in Jenkins but instead use github actions
Updated by ilausuch over 3 years ago
Trying to run the test this in OpenQA. I had always the same issue with the timeouts with the command docker run and podman run.
The problem of these VMs are that we don't have hardware virtualization, so this is really very slow
QEMU: Checking for hardware virtualization : FAIL (Only emulated CPUs are available, performance will be significantly limited)
With @geor helping we are exploring baremetal machines to do this test.
Updated by ilausuch over 3 years ago
The test PR
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/11755
Executed test: http://10.161.229.197/tests/138
Updated by ilausuch over 3 years ago
Initial tests using the openQA repo instead of opensuse
PR: https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/60
Test: http://10.161.229.197/tests/184
Now I am using docker because I have to solve some problems with the podman version resolving the name of the DB
Updated by ilausuch over 3 years ago
I change the status of the PR to be reviewed.
Updated by livdywan over 3 years ago
- Due date changed from 2020-12-24 to 2021-01-22
Updated by ilausuch over 3 years ago
Updated by livdywan over 3 years ago
- Status changed from In Progress to Feedback
Updated by ilausuch over 3 years ago
This test is regular tested in openqa.opensuse.org
https://openqa.opensuse.org/tests/1607610
Updated by okurz over 3 years ago
- Status changed from Resolved to Feedback
The openQA-in-openQA tests are good but I thought we talked about this often enough and decided that we also want at least a subset of tests within the CI tests before merge, see #80520#note-19
Updated by livdywan over 3 years ago
My understanding was, CI didn't work so well, and containers do the job. Otherwise we wouldn't have gone for openQA tests at this point.
Updated by okurz over 3 years ago
cdywan wrote:
"CI didn't work so well, and containers do the job."
Hm, not sure what you mean by that. Maybe you can clarify what you mean. How I see the current situation is like this: With openQA-in-openQA tests we can do real functional tests which is good. However, we can still have more feedback before merge, part of the pull request testing CI tests. In #80520#note-19 I mentioned the options and referenced the os-autoinst pull request that tests container definitions as part of such CI tests. In one of the followup meetings we clarified (I think) that we should also have such tests for openQA. They will not do the same as the openQA-in-openQA tests but check that the Dockerfiles as versioned within our git repo are usable, i.e. buildable.
Updated by livdywan over 3 years ago
okurz wrote:
cdywan wrote:
"CI didn't work so well, and containers do the job."
Hm, not sure what you mean by that. Maybe you can clarify what you mean.
There were problems with the proposed tests on CI, with building and running containers, both with podman and docker, concerning basics like permissions and what user to run as.
How I see the current situation is like this: With openQA-in-openQA tests we can do real functional tests which is good. However, we can still have more feedback before merge, part of the pull request testing CI tests. In #80520#note-19 I mentioned the options and referenced the os-autoinst pull request that tests container definitions as part of such CI tests. In one of the followup meetings we clarified (I think) that we should also have such tests for openQA. They will not do the same as the openQA-in-openQA tests but check that the Dockerfiles as versioned within our git repo are usable, i.e. buildable.
If you're indeed talking about static validation of container files I consider that out of scope for this ticket. The webUI server is started completely without errors
is about running the container successfully.
Updated by okurz over 3 years ago
cdywan wrote:
okurz wrote:
cdywan wrote:
"CI didn't work so well, and containers do the job."
Hm, not sure what you mean by that. Maybe you can clarify what you mean.
There were problems with the proposed tests on CI, with building and running containers, both with podman and docker, concerning basics like permissions and what user to run as.
How I see the current situation is like this: With openQA-in-openQA tests we can do real functional tests which is good. However, we can still have more feedback before merge, part of the pull request testing CI tests. In #80520#note-19 I mentioned the options and referenced the os-autoinst pull request that tests container definitions as part of such CI tests. In one of the followup meetings we clarified (I think) that we should also have such tests for openQA. They will not do the same as the openQA-in-openQA tests but check that the Dockerfiles as versioned within our git repo are usable, i.e. buildable.
If you're indeed talking about static validation of container files I consider that out of scope for this ticket.
The webUI server is started completely without errors
is about running the container successfully.
Well, the "static validation" is merely an implementation consequence when we can not have full functional tests within the PR CI tests.
I think the addition to the openQA-in-openQA tests are great. If you want to split out the mentioned idea into another ticket and call this one done, sure, we can do that. I would just prefer to have some tests for Dockerfiles being executed as part of every PR test.
Updated by livdywan over 3 years ago
okurz wrote:
I think the addition to the openQA-in-openQA tests are great. If you want to split out the mentioned idea into another ticket and call this one done, sure, we can do that. I would just prefer to have some tests for Dockerfiles being executed as part of every PR test.
Fully agree. Between the two of us, I really want to see those static checks, too. I'm just trying to keep ACs in focussed and workable chunks.
I filed #88451
Updated by ilausuch over 3 years ago
- Status changed from Feedback to Resolved
Because of the AC for this ticket is covered and Oliver's suggestions are in the AC of the new ticket, I change the status of this one to resolve.