Project

General

Profile

Actions

action #76978

closed

coordination #127031: [saga][epic] openQA for SUSE customers

coordination #80150: [epic] Scale out openQA: Easier openQA setup

How to run an openQA test in 5 minutes size:M

Added by okurz about 4 years ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-11-04
Due date:
% Done:

0%

Estimated time:

Description

User story

As a potential user of openQA I want to run an openQA test in 5 minutes to be hooked

Acceptance criteria

  • AC1: An openQA instance can be deployed and is used to execute a new (empty) test within 5 minutes (not counting download times on slow connections)
  • AC2: Documentation exists for how to do it

Suggestions

  • Start with existing container solution (and follow the docs)
  • Follow-up with what has been done in #129883
  • Have a workflow (or test or something) that uses the quick bootstrap or comparable approach and runs tests based on github.com/os-autoinst/os-autoinst-distri-example
  • record a video and make some noiz about it :) Either a real life video with explanation how you use it or also something like https://asciinema.org/
  • maybe create/adapt some tool like osc but for openqa that will allow users to run something like osc build

Further details

Inspired by talking to bzoltan :)


Files

Screenshot from 2021-02-10 14-15-01.png (146 KB) Screenshot from 2021-02-10 14-15-01.png ilausuch, 2021-02-10 13:15
openqa-dark.gif (2.29 MB) openqa-dark.gif jbaier_cz, 2024-04-26 19:14
openqa-light.gif (2.33 MB) openqa-light.gif jbaier_cz, 2024-04-26 19:14

Related issues 9 (1 open8 closed)

Related to openQA Project - action #89752: containers: Add a worker service as part of the docker-composeResolvedmkittler2021-03-09

Actions
Related to openQA Project - action #89755: container: Fix missing shared directories and its permissionsResolved2021-03-09

Actions
Related to openQA Project - action #92833: containers: Web UI cannot connect to schedulerResolvedilausuch2021-05-192021-06-04

Actions
Related to openQA Project - coordination #81060: [epic] openQA web UI in kubernetesBlockedokurz2022-05-02

Actions
Related to openQA Project - action #155491: Extend documentation for single-instance-container covering at least how to trigger/clone existing jobs size:SResolvedjbaier_cz2024-01-13

Actions
Blocked by openQA Project - action #89719: docker-compose up fails on masterResolvedilausuch2021-03-09

Actions
Blocked by openQA Project - action #88451: Static validation for containersResolvedokurz2021-02-042021-03-16

Actions
Blocked by openQA Project - action #92893: containers, docker-compose: Ensure that the scheduler can connect to the websockets container size:MResolvedmkittler2021-05-20

Actions
Copied to openQA Project - action #129883: all-in-one openQA container solutionResolvedokurz

Actions
Actions #1

Updated by okurz about 4 years ago

https://github.com/os-autoinst/openQA/pull/3511 as a related first step.

EDIT: merged. Next steps as outlined in suggestions

Actions #2

Updated by okurz about 4 years ago

  • Description updated (diff)
  • Status changed from New to Workable
  • Target version changed from future to Ready
  • Parent task set to #80142
Actions #3

Updated by okurz about 4 years ago

  • Parent task changed from #80142 to #80150
Actions #4

Updated by okurz almost 4 years ago

  • Description updated (diff)
Actions #5

Updated by ilausuch almost 4 years ago

  • Assignee set to ilausuch
Actions #6

Updated by openqa_review almost 4 years ago

  • Due date set to 2021-02-13

Setting due date based on mean cycle time of SUSE QE Tools

Actions #7

Updated by ilausuch almost 4 years ago

I recorded a video to show how to deploy openqa_webui container

https://asciinema.org/a/x8UYkEysNsGlAfjCamvzLmjh0

git clone https://github.com/os-autoinst/openQA.git

mkdir -p data/factory/{iso,hdd,other,tmp} data/{testresults,tests,conf} data/certs/{ssl.crt,ssl.key}
cp openQA/container/openqa_data/data.template/conf/* data/conf
cp openQA/container/webui/conf/database.ini data/conf/
cat /tmp/openqa.ini | sed 's/method = OpenID/method = Fake/' >data/conf/openqa.ini

docker network create openQA

docker run -d --network openQA -e POSTGRES_PASSWORD=openqa -e POSTGRES_USER=openqa -e POSTGRES_DB=openqa --net-alias=db --name openqa_db postgres
docker logs -f openqa_db

openssl req -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -subj '/CN=www.mydom.com/O=My Company Name LTD./C=DE' -out data/certs/ssl.crt/server.crt -keyout data/certs/ssl.key/server.key
cp data/certs/ssl.crt/server.crt data/certs/ssl.crt/ca.crt

volumes="-v $(pwd)/data:/data"
certificates="-v $(pwd)/data/certs/ssl.crt:/etc/apache2/ssl.crt -v $(pwd)/data/certs/ssl.key:/etc/apache2/ssl.key"

docker build openQA/container/webui -t openqa_webui
docker run -d --network openQA $volumes $certificates -p 80:80 --net-alias=openqa_webui --name openqa_webui openqa_webui
docker logs -f openqa_webui
Actions #8

Updated by okurz almost 4 years ago

To have videos is great but we should not encourage users to use the development repos. Also our goal should be "single-command" solutions. As openqa-bootstrap already supports to take jobs to clone as parameters I suggest to start with that

Actions #9

Updated by livdywan almost 4 years ago

This seems very similar to what openqa-bootstrap-container does except that it uses docker as a runtime. Maybe we want that as a re-usable script?

The video teaches the user how to build and deploy openQA, and that's what this ticket asks for, so I like it. Not sure if this is exactly what I was expecting since the audience wasn't defined. Deplying a container probably makes it very widely accessible so I like that. My main concern would be the use of the git repo as already mentioned by @okurz... thanks to #43718 we presumably have versioned images that can be used here and avoid git?

Actions #10

Updated by livdywan almost 4 years ago

  • Status changed from Workable to In Progress
Actions #11

Updated by okurz almost 4 years ago

cdywan wrote:

This seems very similar to what openqa-bootstrap-container does except that it uses docker as a runtime. Maybe we want that as a re-usable script?

Yes, that was mentioned as the second point within the suggestions.

The video teaches the user how to build and deploy openQA, and that's what this ticket asks for, so I like it. Not sure if this is exactly what I was expecting since the audience wasn't defined.

Well, the user story has an actor "potential user of openQA" and the goal is "to be hooked" :)

And yes, a container based solution was suggested. That is fine. But as we already have an approach as documented in http://open.qa/docs/#bootstrapping that shows how an openQA installation can be installed and immediately start a clone-job I consider the multiple steps shown in the video as a step backwards.

Actions #12

Updated by livdywan almost 4 years ago

okurz wrote:

Well, the user story has an actor "potential user of openQA" and the goal is "to be hooked" :)

And yes, a container based solution was suggested. That is fine. But as we already have an approach as documented in http://open.qa/docs/#bootstrapping that shows how an openQA installation can be installed and immediately start a clone-job I consider the multiple steps shown in the video as a step backwards.

Those instructions say, here's a command, go read a file to figure out how to actually use it. I think that's what we want to improve here.

  1. What concrete steps are missing here in the typical case?
  2. Is it desirable to customize the container like in #76978#note-7
  3. Certificate setup like in #76978#note-7
  4. Running an actual test, not covered yet
Actions #13

Updated by kraih almost 4 years ago

Might be fun to piggyback on k3s for this. https://k3s.io Then the user would need literally nothing but a Linux kernel and Bash to get started with openQA. You might not like curl | bash one-liners for one reason or another, but that's what gets people hooked.

Actions #14

Updated by okurz almost 4 years ago

exactly :) That makes sense.

Actions #15

Updated by livdywan almost 4 years ago

@ilausuch Take a look at openqa-bootstrap{,-container for inspiration

  • Avoid git repos - keep in mind the new user who just wants to run a test
  • Pretend there's nothing installed, don't assume Tumbleweed/SLE (we cover that use case with existing scripts, no need to replicate that)
  • Use --device=/dev/kvm
  • Keep it simple, one or two lines if possible
  • Make sure any setup of e.g. qemu/isotovideo is inside the container
Actions #16

Updated by ilausuch almost 4 years ago

Thanks for all your help.
I recorded a video showing how to setup the worker https://asciinema.org/a/390320

The script to have the worker running is this:

$ chmod a+w data/testresults

In this section we are going to use only opensuse test suites, but if you want more add them following the example:

$ git clone git@github.com:os-autoinst/os-autoinst-distri-opensuse.git

$ ln -s os-autoinst-distri-opensuse opensuse

$ git clone https://github.com/os-autoinst/os-autoinst-needles-opensuse.git

$ mv os-autoinst-needles-opensuse/ opensuse/products/opensuse/needles

Create a key/secret pair on your webUI and replace the example ones to configure the client.conf

$ echo -e “[openqa_webui]\nkey = 5AD3533BB586FD54\nsecret = AE0A2A27FD82D530” > data/conf/client.conf

Ensure that the variable volumes is set

$ volumes=”-v $(pwd)/data:/data”

Build the worker

$ docker build openQA/container/worker -t openqa_worker

Run the worker

$ docker run -d —network openQA $volumes —device=/dev/kvm —privileged —name openqa_worker openqa_worker

You can launch a test accessing to the web UI

$ docker exec -ti openqa_webui /usr/share/openqa/script/clone_job.pl —apikey 5AD3533BB586FD54 —apisecret AE0A2A27FD82D530 https://openqa.opensuse.org/tests/1620808
Actions #17

Updated by ilausuch almost 4 years ago

Trying to figure out how to run the worker without being root

The group of the kvm is 128 that doesn't belongs to any group in the container

4a8e60738367:/ # ls -lh /dev/kvm 
crw-rw---- 1 root 128 10, 232 Feb 11 08:34 /dev/kvm

I tried to assign the group using the number, but doesn't work

usermod -g 128 _openqa-worker
usermod: group '128' does not exist
Actions #18

Updated by ilausuch almost 4 years ago

thanks George we found a solution using groupmod

groupmod -g 128 kvm
Actions #19

Updated by ilausuch almost 4 years ago

cdywan wrote:

@ilausuch Take a look at openqa-bootstrap{,-container for inspiration

  • Avoid git repos - keep in mind the new user who just wants to run a test

I have this in mind, the previous recordings were tests for the final recording that will be using the container repository

  • Pretend there's nothing installed, don't assume Tumbleweed/SLE (we cover that use case with existing scripts, no need to replicate that)

Right now, these steps are valid for in a Ubuntu too. So we could consider that is independent to the linux distro. Only, the user has to install docker and as far as I know kvm too

  • Use --device=/dev/kvm

Yes, the PR associated to this point is this https://github.com/os-autoinst/openQA/pull/3715

  • Keep it simple, one or two lines if possible

There are several steps we cannot avoid, because we have different containers, in this case a progress, webUI and worker has to be deployed.

Additionally some aspects like volumes and certificates has to be provided from the user.

  • Volumes: We need to set-up an initial directory structure and provide to the containers via volumes. For instance, the configuration on webui and worker container doesn't exists. Has to be provided.
  • Certificates: In some previous discussions we agree that the certificates has to be provided by the user.
  • Tests: If we are working a generic tool the tests and needles cannot be part of the worker container image because we don't know what tests the user is going to use. Then has to be downloaded and put in the right directory by the user

Update: I am preparing a simple script to help the user to run all of these in two or there lines

  • Make sure any setup of e.g. qemu/isotovideo is inside the container

That works. The video of the job is stored on assets can can be played.

Actions #20

Updated by livdywan almost 4 years ago

  • Due date changed from 2021-02-13 to 2021-02-19

Bumping the Due date since enabling kvm isn't fully solved yet and there were open questions with scripting installation. See the record of the last conversation above.

Actions #21

Updated by livdywan almost 4 years ago

  • Due date changed from 2021-02-19 to 2021-02-26

Update: I am preparing a simple script to help the user to run all of these in two or there lines

https://github.com/os-autoinst/openQA/pull/3737

Actions #22

Updated by livdywan almost 4 years ago

ilausuch wrote:

  • Keep it simple, one or two lines if possible

There are several steps we cannot avoid, because we have different containers, in this case a progress, webUI and worker has to be deployed.

Additionally some aspects like volumes and certificates has to be provided from the user.

  • Volumes: We need to set-up an initial directory structure and provide to the containers via volumes. For instance, the configuration on webui and worker container doesn't exists. Has to be provided.
  • Certificates: In some previous discussions we agree that the certificates has to be provided by the user.
  • Tests: If we are working a generic tool the tests and needles cannot be part of the worker container image because we don't know what tests the user is going to use. Then has to be downloaded and put in the right directory by the user

Update: I am preparing a simple script to help the user to run all of these in two or there lines

I commented on the PR but for the record also replying here.

Do we really want a whole CLI here? Can't we use docker-compose for the setup of volumes, different containers etc.?

I could see if letting users supply certificates or openSUSE test repos is not trivial, but then that's the only things the script should be doing.

Actions #23

Updated by ilausuch almost 4 years ago

Working on the docker-compose file
Some considerations:

  • Because all parts work in different containers, the OPENQA_SCHEDULER_HOST and OPENQA_WEB_SOCKETS_HOST env variables need to be set in the webui containers and the scheduler so that when they communicate properly
  • The directory factory needs the subdirectories: iso,hdd,other,tmp
  • The directories factory and subdirectories, and testresults has to be read-write
  • To add a worker in the docker-compose a API key/secret is needed from the beginning
  • The scheduler, webui, and others tries to create the tables on the database at the same time creating failures. Is needed to delay some of them to prevent this.

Preparing the PR. Now It can execute jobs, but doesn't show anything in the live view.

Actions #24

Updated by livdywan almost 4 years ago

ilausuch wrote:

  • The directory factory needs the subdirectories: iso,hdd,other,tmp
  • The directories factory and subdirectories, and testresults has to be read-write

Could we have those created on demand by the worker? It occurs to me this is something I'd also do as part of a development setup as well, ensure e.g. share/factory/hdd exists because otherwise cloning won't work. Just empty folders are expected here. 🤔

Actions #25

Updated by ilausuch almost 4 years ago

  • Due date changed from 2021-02-26 to 2021-03-05
Actions #26

Updated by ilausuch almost 4 years ago

cdywan wrote:

ilausuch wrote:

  • The directory factory needs the subdirectories: iso,hdd,other,tmp
  • The directories factory and subdirectories, and testresults has to be read-write

Could we have those created on demand by the worker? It occurs to me this is something I'd also do as part of a development setup as well, ensure e.g. share/factory/hdd exists because otherwise cloning won't work. Just empty folders are expected here. 🤔

This was my first approach, but it's more easy than that. The docker-compose can do it. I've just checked that work

Actions #27

Updated by ilausuch almost 4 years ago

Created the PR that uses the docker-container to deploy the entire openQA system
https://github.com/os-autoinst/openQA/pull/3755

Considerations:

  • The user must install the tests on workdir/tests
  • To run opensuse and other SUSE tests execute: docker exec -t openqa_worker zypper in -y os-autoinst-distri-opensuse-deps
Actions #28

Updated by livdywan over 3 years ago

  • Blocked by action #89719: docker-compose up fails on master added
Actions #29

Updated by okurz over 3 years ago

  • Status changed from In Progress to Blocked

We discussed in the daily 2021-03-09 that we encountered multiple issues with the current docker-compose.yaml file so that the current docker-compose.yaml which we should cover in individual tickets so that we can progress within cycle times. Also we need an automatic check for docker-compose, see #89722

Actions #30

Updated by okurz over 3 years ago

  • Due date deleted (2021-03-05)
Actions #31

Updated by livdywan over 3 years ago

  • Blocked by action #89731: containers: The deploy using docker-compose is not stable and eventually fails added
Actions #32

Updated by livdywan over 3 years ago

  • Blocked by deleted (action #89731: containers: The deploy using docker-compose is not stable and eventually fails )
Actions #33

Updated by okurz over 3 years ago

  • Blocked by action #88451: Static validation for containers added
Actions #34

Updated by livdywan over 3 years ago

  • Related to action #89752: containers: Add a worker service as part of the docker-compose added
Actions #35

Updated by livdywan over 3 years ago

  • Related to action #89755: container: Fix missing shared directories and its permissions added
Actions #36

Updated by ilausuch over 3 years ago

  • Status changed from Blocked to In Progress

I unblocked because recent related tickets are closed and with what we have now it could be possible. I am going to explore if is it possible now run a full env with the web UI using the docker-compose and worker using a few commands. If is not possible I'll write down what are the reasons and then create new tickets if are necessary

Actions #37

Updated by ilausuch over 3 years ago

  • Status changed from In Progress to Blocked

During this new attempt to run a job I found a problem:

webui_2          | [warn] Unable to wakeup scheduler: Connection refused

I tested the access to the scheduler from this webui and works

$ ping scheduler
PING scheduler (192.168.80.8) 56(84) bytes of data.
64 bytes from webui_scheduler_1.webui_default (192.168.80.8): icmp_seq=1 ttl=64 time=0.126 ms

$ telnet scheduler 9529
Trying 192.168.80.8...
Connected to scheduler.
Escape character is '^]'.

So, maybe is a configuration problem or other env variables needed

To prepare the environment:

$ cd openQA/containers/webui
$ docker-compose build
# Edit openqa.ini and set auth method to Fake
$ docker-compose up -d
# Now you can access to the UI at http://localhost:9526 and login. If you want you can create a new credentials, there are ones by default

# go to the directory workdir/data/tests and download the test suites you want, for instance opensuse
$ sudo git clone https://github.com/os-autoinst/os-autoinst-distri-opensuse.git
$ sudo mv os-autoinst-distri-opensuse opensuse
$ sudo git clone https://github.com/os-autoinst/os-autoinst-needles-opensuse.git
$ sudo mv os-autoinst-needles-opensuse/ opensuse/products/opensuse/needles

# Move to the worker directory
$ ln -s ../webui/workdir/
# Edit conf/worker.ini and change HOST to http://nginx:9526
# Edit conf/client.conf and change [openqa_webui] to [nginx]

$ docker run -d --network=webui_default -v $(pwd)/workdir/data/factory:/data/factory:rw -v $(pwd)/workdir/data/tests:/data/tests:ro -v $(pwd)/conf:/data/conf:ro --device=/dev/kvm --privileged openqa_worker --name openqa_worker

# Check that the worker has connected 
$ docker logs -f openqa_worker

To launch the test:

$ docker-compose exec webui bash
$ /usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host http://nginx:9526 1738950
# This download all the assets correctly

I created a ticket to solve this problem #92833 and this one will be blocked until the other is solved

Actions #38

Updated by ilausuch over 3 years ago

  • Related to action #92833: containers: Web UI cannot connect to scheduler added
Actions #39

Updated by ilausuch over 3 years ago

  • Blocked by action #92893: containers, docker-compose: Ensure that the scheduler can connect to the websockets container size:M added
Actions #40

Updated by okurz over 3 years ago

Actions #41

Updated by ilausuch over 3 years ago

  • Assignee deleted (ilausuch)
Actions #42

Updated by okurz over 3 years ago

  • Assignee set to okurz

Blocked should always have assignee

Actions #43

Updated by okurz about 3 years ago

  • Status changed from Blocked to New
  • Assignee deleted (okurz)

blocker #92893 resolved.

Actions #44

Updated by okurz about 3 years ago

  • Target version changed from Ready to future
Actions #45

Updated by okurz over 1 year ago

  • Target version changed from future to Ready
Actions #46

Updated by okurz over 1 year ago

  • Copied to action #129883: all-in-one openQA container solution added
Actions #47

Updated by dheidler over 1 year ago

  • Description updated (diff)
Actions #48

Updated by livdywan over 1 year ago

  • Status changed from New to Blocked
  • Assignee set to okurz

Blocking on #129883

Actions #49

Updated by okurz over 1 year ago

  • Description updated (diff)
  • Status changed from Blocked to New
  • Assignee deleted (okurz)
Actions #50

Updated by okurz over 1 year ago

  • Subject changed from How to run an openQA test in 5 minutes to How to run an openQA test in 5 minutes size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #51

Updated by tinita over 1 year ago

  • Target version changed from Ready to future

Setting to future for now because our backlog is full

Actions #52

Updated by okurz 8 months ago

  • Target version changed from future to Ready
Actions #53

Updated by jbaier_cz 7 months ago

  • Assignee set to jbaier_cz
Actions #54

Updated by jbaier_cz 7 months ago

  • Status changed from Workable to In Progress
Actions #55

Updated by openqa_review 7 months ago

  • Due date set to 2024-05-10

Setting due date based on mean cycle time of SUSE QE Tools

Actions #56

Updated by jbaier_cz 7 months ago

A very simple way for cloning a custom job from other openQA instance can be just something like:

podman run --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -it registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman exec openqa /usr/share/openqa/script/fetchneedles
podman exec openqa openqa-clone-job https://openqa.opensuse.org/tests/4109159
podman exec openqa openqa-cli monitor 1
podman exec openqa openqa-cli api --pretty jobs/1

The full "run one job locally" equivalent from o3/osd could be made even simpler as we have some built-in support. The full workflow can be summed up into:

podman run --detach --env skip_suse_tests="" --env skip_suse_specifics="" --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -ti registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman wait --condition healthy openqa
podman exec openqa openqa-clone-job https://openqa.opensuse.org/tests/4109159
podman exec openqa openqa-cli monitor 1
podman exec openqa openqa-cli api --pretty jobs/1
podman stop openqa

You can of course do something more "wild" and schedule a custom job as we did as a part of openqa-in-openqa test:

podman run --detach --env skip_suse_tests="" --env skip_suse_specifics="" --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -ti registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman wait --condition healthy openqa
wget -O scenario-definitions.yaml https://raw.githubusercontent.com/os-autoinst/os-autoinst-distri-example/main/scenario-definitions.yaml
podman cp scenario-definitions.yaml openqa:scenario-definitions.yaml
podman exec -ti openqa openqa-cli schedule --monitor --param-file SCENARIO_DEFINITIONS_YAML=scenario-definitions.yaml DISTRI=example VERSION=0 FLAVOR=DVD ARCH=x86_64 TEST=simple_boot _GROUP_ID=0 BUILD=test CASEDIR=https://github.com/os-autoinst/os-autoinst-distri-example.git NEEDLES_DIR=%%CASEDIR%%/needles
podman exec openqa openqa-cli api jobs/1 | jq -r .job.result
podman stop openqa

I will enhance the documentation to have those examples and create a short "video" featuring this.

Updated by jbaier_cz 7 months ago

Not sure if we like light or dark theme more, but this could be the first proof-of-concept. As a next step I will prepare some PR into our documentation.

Actions #58

Updated by jbaier_cz 7 months ago · Edited

  • Status changed from In Progress to Blocked

Because generating documentation locally seems not to be well documented, let's sort out #159747 first.

Actions #59

Updated by jbaier_cz 7 months ago

  • Related to action #155491: Extend documentation for single-instance-container covering at least how to trigger/clone existing jobs size:S added
Actions #60

Updated by jbaier_cz 7 months ago

With #159747 almost out of the way, I created https://github.com/os-autoinst/openQA/pull/5611 for including a short demo inside our documentation. There is also an automated way to regenerate the image if needed and to create more if wanted. This might also satisfy the conditions for #155491. If needed, I can also include the information from the recording in a written form inside the same documentation section.

Actions #61

Updated by jbaier_cz 7 months ago

  • Due date deleted (2024-05-10)
  • Status changed from Blocked to Resolved
Actions

Also available in: Atom PDF