Project

General

Profile

action #76978

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

coordination #80150: [epic] Scale out openQA: Easier openQA setup

How to run an openQA test in 5 minutes

Added by okurz 9 months ago. Updated 2 months ago.

Status:
Blocked
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-11-04
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

User story

As a potential user of openQA I want to run an openQA test in 5 minutes to be hooked

Acceptance criteria

  • AC1: An openQA instance can be installed and is used to execute a new (empty) test within 5 minutes (not counting download times on slow connections)

Suggestions

  • In your local openQA working copy, try out to build and run containers, e.g. podman build -t openqa-webui docker/webui and then podman run --rm -it openqa-webui
  • We already have a quick bootstrap method but it might not be prominent enough and we can present it as a "first-class citizen"
  • Have a workflow (or test or something) that uses the quick bootstrap or comparable approach and runs tests based on github.com/os-autoinst/os-autoinst-distri-example
  • record a video and make some noiz about it :)

Further details

Inspired by talking to bzoltan :)


Related issues

Related to openQA Project - action #89752: containers: Add a worker service as part of the docker-composeWorkable2021-03-09

Related to openQA Project - action #89755: container: Fix missing shared directories and its permissionsResolved2021-03-09

Related to openQA Project - action #92833: containers: Web UI cannot connect to schedulerResolved2021-05-192021-06-04

Related to openQA Project - action #81060: Create a helm chart to deploy web UI in kubernetesBlocked2020-12-14

Blocked by openQA Project - action #89719: docker-compose up fails on masterResolved2021-03-09

Blocked by openQA Project - action #88451: Static validation for containersResolved2021-02-042021-03-16

Blocked by openQA Project - action #92893: containers, docker-compose: Ensure that the scheduler can connect to the websockets containerNew2021-05-20

History

#1 Updated by okurz 9 months ago

https://github.com/os-autoinst/openQA/pull/3511 as a related first step.

EDIT: merged. Next steps as outlined in suggestions

#2 Updated by okurz 8 months ago

  • Description updated (diff)
  • Status changed from New to Workable
  • Target version changed from future to Ready
  • Parent task set to #80142

#3 Updated by okurz 8 months ago

  • Parent task changed from #80142 to #80150

#4 Updated by okurz 8 months ago

  • Description updated (diff)

#5 Updated by ilausuch 6 months ago

  • Assignee set to ilausuch

#6 Updated by openqa_review 6 months ago

  • Due date set to 2021-02-13

Setting due date based on mean cycle time of SUSE QE Tools

#7 Updated by ilausuch 6 months ago

I recorded a video to show how to deploy openqa_webui container

https://asciinema.org/a/x8UYkEysNsGlAfjCamvzLmjh0

git clone https://github.com/os-autoinst/openQA.git

mkdir -p data/factory/{iso,hdd,other,tmp} data/{testresults,tests,conf} data/certs/{ssl.crt,ssl.key}
cp openQA/container/openqa_data/data.template/conf/* data/conf
cp openQA/container/webui/conf/database.ini data/conf/
cat /tmp/openqa.ini | sed 's/method = OpenID/method = Fake/' >data/conf/openqa.ini

docker network create openQA

docker run -d --network openQA -e POSTGRES_PASSWORD=openqa -e POSTGRES_USER=openqa -e POSTGRES_DB=openqa --net-alias=db --name openqa_db postgres
docker logs -f openqa_db

openssl req -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -subj '/CN=www.mydom.com/O=My Company Name LTD./C=DE' -out data/certs/ssl.crt/server.crt -keyout data/certs/ssl.key/server.key
cp data/certs/ssl.crt/server.crt data/certs/ssl.crt/ca.crt

volumes="-v $(pwd)/data:/data"
certificates="-v $(pwd)/data/certs/ssl.crt:/etc/apache2/ssl.crt -v $(pwd)/data/certs/ssl.key:/etc/apache2/ssl.key"

docker build openQA/container/webui -t openqa_webui
docker run -d --network openQA $volumes $certificates -p 80:80 --net-alias=openqa_webui --name openqa_webui openqa_webui
docker logs -f openqa_webui

#8 Updated by okurz 6 months ago

To have videos is great but we should not encourage users to use the development repos. Also our goal should be "single-command" solutions. As openqa-bootstrap already supports to take jobs to clone as parameters I suggest to start with that

#9 Updated by cdywan 6 months ago

This seems very similar to what openqa-bootstrap-container does except that it uses docker as a runtime. Maybe we want that as a re-usable script?

The video teaches the user how to build and deploy openQA, and that's what this ticket asks for, so I like it. Not sure if this is exactly what I was expecting since the audience wasn't defined. Deplying a container probably makes it very widely accessible so I like that. My main concern would be the use of the git repo as already mentioned by @okurz... thanks to #43718 we presumably have versioned images that can be used here and avoid git?

#10 Updated by cdywan 6 months ago

  • Status changed from Workable to In Progress

#11 Updated by okurz 6 months ago

cdywan wrote:

This seems very similar to what openqa-bootstrap-container does except that it uses docker as a runtime. Maybe we want that as a re-usable script?

Yes, that was mentioned as the second point within the suggestions.

The video teaches the user how to build and deploy openQA, and that's what this ticket asks for, so I like it. Not sure if this is exactly what I was expecting since the audience wasn't defined.

Well, the user story has an actor "potential user of openQA" and the goal is "to be hooked" :)

And yes, a container based solution was suggested. That is fine. But as we already have an approach as documented in http://open.qa/docs/#bootstrapping that shows how an openQA installation can be installed and immediately start a clone-job I consider the multiple steps shown in the video as a step backwards.

#12 Updated by cdywan 6 months ago

okurz wrote:

Well, the user story has an actor "potential user of openQA" and the goal is "to be hooked" :)

And yes, a container based solution was suggested. That is fine. But as we already have an approach as documented in http://open.qa/docs/#bootstrapping that shows how an openQA installation can be installed and immediately start a clone-job I consider the multiple steps shown in the video as a step backwards.

Those instructions say, here's a command, go read a file to figure out how to actually use it. I think that's what we want to improve here.

  1. What concrete steps are missing here in the typical case?
  2. Is it desirable to customize the container like in #76978#note-7
  3. Certificate setup like in #76978#note-7
  4. Running an actual test, not covered yet

#13 Updated by kraih 6 months ago

Might be fun to piggyback on k3s for this. https://k3s.io Then the user would need literally nothing but a Linux kernel and Bash to get started with openQA. You might not like curl | bash one-liners for one reason or another, but that's what gets people hooked.

#14 Updated by okurz 6 months ago

exactly :) That makes sense.

#15 Updated by cdywan 6 months ago

ilausuch Take a look at openqa-bootstrap{,-container for inspiration

  • Avoid git repos - keep in mind the new user who just wants to run a test
  • Pretend there's nothing installed, don't assume Tumbleweed/SLE (we cover that use case with existing scripts, no need to replicate that)
  • Use --device=/dev/kvm
  • Keep it simple, one or two lines if possible
  • Make sure any setup of e.g. qemu/isotovideo is inside the container

#16 Updated by ilausuch 6 months ago

10983

Thanks for all your help.
I recorded a video showing how to setup the worker https://asciinema.org/a/390320

The script to have the worker running is this:

$ chmod a+w data/testresults

In this section we are going to use only opensuse test suites, but if you want more add them following the example:

$ git clone git@github.com:os-autoinst/os-autoinst-distri-opensuse.git

$ ln -s os-autoinst-distri-opensuse opensuse

$ git clone https://github.com/os-autoinst/os-autoinst-needles-opensuse.git

$ mv os-autoinst-needles-opensuse/ opensuse/products/opensuse/needles

Create a key/secret pair on your webUI and replace the example ones to configure the client.conf

$ echo -e “[openqa_webui]\nkey = 5AD3533BB586FD54\nsecret = AE0A2A27FD82D530” > data/conf/client.conf

Ensure that the variable volumes is set

$ volumes=”-v $(pwd)/data:/data”

Build the worker

$ docker build openQA/container/worker -t openqa_worker

Run the worker

$ docker run -d —network openQA $volumes —device=/dev/kvm —privileged —name openqa_worker openqa_worker

You can launch a test accessing to the web UI

$ docker exec -ti openqa_webui /usr/share/openqa/script/clone_job.pl —apikey 5AD3533BB586FD54 —apisecret AE0A2A27FD82D530 https://openqa.opensuse.org/tests/1620808

#17 Updated by ilausuch 6 months ago

Trying to figure out how to run the worker without being root

The group of the kvm is 128 that doesn't belongs to any group in the container

4a8e60738367:/ # ls -lh /dev/kvm 
crw-rw---- 1 root 128 10, 232 Feb 11 08:34 /dev/kvm

I tried to assign the group using the number, but doesn't work

usermod -g 128 _openqa-worker
usermod: group '128' does not exist

#18 Updated by ilausuch 6 months ago

thanks George we found a solution using groupmod

groupmod -g 128 kvm

#19 Updated by ilausuch 5 months ago

cdywan wrote:

ilausuch Take a look at openqa-bootstrap{,-container for inspiration

  • Avoid git repos - keep in mind the new user who just wants to run a test

I have this in mind, the previous recordings were tests for the final recording that will be using the container repository

  • Pretend there's nothing installed, don't assume Tumbleweed/SLE (we cover that use case with existing scripts, no need to replicate that)

Right now, these steps are valid for in a Ubuntu too. So we could consider that is independent to the linux distro. Only, the user has to install docker and as far as I know kvm too

  • Use --device=/dev/kvm

Yes, the PR associated to this point is this https://github.com/os-autoinst/openQA/pull/3715

  • Keep it simple, one or two lines if possible

There are several steps we cannot avoid, because we have different containers, in this case a progress, webUI and worker has to be deployed.

Additionally some aspects like volumes and certificates has to be provided from the user.

  • Volumes: We need to set-up an initial directory structure and provide to the containers via volumes. For instance, the configuration on webui and worker container doesn't exists. Has to be provided.
  • Certificates: In some previous discussions we agree that the certificates has to be provided by the user.
  • Tests: If we are working a generic tool the tests and needles cannot be part of the worker container image because we don't know what tests the user is going to use. Then has to be downloaded and put in the right directory by the user

Update: I am preparing a simple script to help the user to run all of these in two or there lines

  • Make sure any setup of e.g. qemu/isotovideo is inside the container

That works. The video of the job is stored on assets can can be played.

#20 Updated by cdywan 5 months ago

  • Due date changed from 2021-02-13 to 2021-02-19

Bumping the Due date since enabling kvm isn't fully solved yet and there were open questions with scripting installation. See the record of the last conversation above.

#21 Updated by cdywan 5 months ago

  • Due date changed from 2021-02-19 to 2021-02-26

Update: I am preparing a simple script to help the user to run all of these in two or there lines

https://github.com/os-autoinst/openQA/pull/3737

#22 Updated by cdywan 5 months ago

ilausuch wrote:

  • Keep it simple, one or two lines if possible

There are several steps we cannot avoid, because we have different containers, in this case a progress, webUI and worker has to be deployed.

Additionally some aspects like volumes and certificates has to be provided from the user.

  • Volumes: We need to set-up an initial directory structure and provide to the containers via volumes. For instance, the configuration on webui and worker container doesn't exists. Has to be provided.
  • Certificates: In some previous discussions we agree that the certificates has to be provided by the user.
  • Tests: If we are working a generic tool the tests and needles cannot be part of the worker container image because we don't know what tests the user is going to use. Then has to be downloaded and put in the right directory by the user

Update: I am preparing a simple script to help the user to run all of these in two or there lines

I commented on the PR but for the record also replying here.

Do we really want a whole CLI here? Can't we use docker-compose for the setup of volumes, different containers etc.?

I could see if letting users supply certificates or openSUSE test repos is not trivial, but then that's the only things the script should be doing.

#23 Updated by ilausuch 5 months ago

Working on the docker-compose file
Some considerations:

  • Because all parts work in different containers, the OPENQA_SCHEDULER_HOST and OPENQA_WEB_SOCKETS_HOST env variables need to be set in the webui containers and the scheduler so that when they communicate properly
  • The directory factory needs the subdirectories: iso,hdd,other,tmp
  • The directories factory and subdirectories, and testresults has to be read-write
  • To add a worker in the docker-compose a API key/secret is needed from the beginning
  • The scheduler, webui, and others tries to create the tables on the database at the same time creating failures. Is needed to delay some of them to prevent this.

Preparing the PR. Now It can execute jobs, but doesn't show anything in the live view.

#24 Updated by cdywan 5 months ago

ilausuch wrote:

  • The directory factory needs the subdirectories: iso,hdd,other,tmp
  • The directories factory and subdirectories, and testresults has to be read-write

Could we have those created on demand by the worker? It occurs to me this is something I'd also do as part of a development setup as well, ensure e.g. share/factory/hdd exists because otherwise cloning won't work. Just empty folders are expected here. 🤔

#25 Updated by ilausuch 5 months ago

  • Due date changed from 2021-02-26 to 2021-03-05

#26 Updated by ilausuch 5 months ago

cdywan wrote:

ilausuch wrote:

  • The directory factory needs the subdirectories: iso,hdd,other,tmp
  • The directories factory and subdirectories, and testresults has to be read-write

Could we have those created on demand by the worker? It occurs to me this is something I'd also do as part of a development setup as well, ensure e.g. share/factory/hdd exists because otherwise cloning won't work. Just empty folders are expected here. 🤔

This was my first approach, but it's more easy than that. The docker-compose can do it. I've just checked that work

#27 Updated by ilausuch 5 months ago

Created the PR that uses the docker-container to deploy the entire openQA system
https://github.com/os-autoinst/openQA/pull/3755

Considerations:

  • The user must install the tests on workdir/tests
  • To run opensuse and other SUSE tests execute: docker exec -t openqa_worker zypper in -y os-autoinst-distri-opensuse-deps

#28 Updated by cdywan 5 months ago

  • Blocked by action #89719: docker-compose up fails on master added

#29 Updated by okurz 5 months ago

  • Status changed from In Progress to Blocked

We discussed in the daily 2021-03-09 that we encountered multiple issues with the current docker-compose.yaml file so that the current docker-compose.yaml which we should cover in individual tickets so that we can progress within cycle times. Also we need an automatic check for docker-compose, see #89722

#30 Updated by okurz 5 months ago

  • Due date deleted (2021-03-05)

#31 Updated by cdywan 5 months ago

  • Blocked by action #89731: containers: The deploy using docker-compose is not stable and eventually fails added

#32 Updated by cdywan 5 months ago

  • Blocked by deleted (action #89731: containers: The deploy using docker-compose is not stable and eventually fails )

#33 Updated by okurz 5 months ago

  • Blocked by action #88451: Static validation for containers added

#34 Updated by cdywan 5 months ago

  • Related to action #89752: containers: Add a worker service as part of the docker-compose added

#35 Updated by cdywan 5 months ago

  • Related to action #89755: container: Fix missing shared directories and its permissions added

#36 Updated by ilausuch 2 months ago

  • Status changed from Blocked to In Progress

I unblocked because recent related tickets are closed and with what we have now it could be possible. I am going to explore if is it possible now run a full env with the web UI using the docker-compose and worker using a few commands. If is not possible I'll write down what are the reasons and then create new tickets if are necessary

#37 Updated by ilausuch 2 months ago

  • Status changed from In Progress to Blocked

During this new attempt to run a job I found a problem:

webui_2          | [warn] Unable to wakeup scheduler: Connection refused

I tested the access to the scheduler from this webui and works

$ ping scheduler
PING scheduler (192.168.80.8) 56(84) bytes of data.
64 bytes from webui_scheduler_1.webui_default (192.168.80.8): icmp_seq=1 ttl=64 time=0.126 ms

$ telnet scheduler 9529
Trying 192.168.80.8...
Connected to scheduler.
Escape character is '^]'.

So, maybe is a configuration problem or other env variables needed

To prepare the environment:

$ cd openQA/containers/webui
$ docker-compose build
# Edit openqa.ini and set auth method to Fake
$ docker-compose up -d
# Now you can access to the UI at http://localhost:9526 and login. If you want you can create a new credentials, there are ones by default

# go to the directory workdir/data/tests and download the test suites you want, for instance opensuse
$ sudo git clone https://github.com/os-autoinst/os-autoinst-distri-opensuse.git
$ sudo mv os-autoinst-distri-opensuse opensuse
$ sudo git clone https://github.com/os-autoinst/os-autoinst-needles-opensuse.git
$ sudo mv os-autoinst-needles-opensuse/ opensuse/products/opensuse/needles

# Move to the worker directory
$ ln -s ../webui/workdir/
# Edit conf/worker.ini and change HOST to http://nginx:9526
# Edit conf/client.conf and change [openqa_webui] to [nginx]

$ docker run -d --network=webui_default -v $(pwd)/workdir/data/factory:/data/factory:rw -v $(pwd)/workdir/data/tests:/data/tests:ro -v $(pwd)/conf:/data/conf:ro --device=/dev/kvm --privileged openqa_worker --name openqa_worker

# Check that the worker has connected 
$ docker logs -f openqa_worker

To launch the test:

$ docker-compose exec webui bash
$ /usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host http://nginx:9526 1738950
# This download all the assets correctly

I created a ticket to solve this problem #92833 and this one will be blocked until the other is solved

#38 Updated by ilausuch 2 months ago

  • Related to action #92833: containers: Web UI cannot connect to scheduler added

#39 Updated by ilausuch 2 months ago

  • Blocked by action #92893: containers, docker-compose: Ensure that the scheduler can connect to the websockets container added

#40 Updated by okurz about 1 month ago

  • Related to action #81060: Create a helm chart to deploy web UI in kubernetes added

Also available in: Atom PDF