QA » openQA Project

https://github.com/os-autoinst/openQA/pull/3511 as a related first step.

EDIT: merged. Next steps as outlined in suggestions

I recorded a video to show how to deploy openqa_webui container

https://asciinema.org/a/x8UYkEysNsGlAfjCamvzLmjh0

git clone https://github.com/os-autoinst/openQA.git

mkdir -p data/factory/{iso,hdd,other,tmp} data/{testresults,tests,conf} data/certs/{ssl.crt,ssl.key}
cp openQA/container/openqa_data/data.template/conf/* data/conf
cp openQA/container/webui/conf/database.ini data/conf/
cat /tmp/openqa.ini | sed 's/method = OpenID/method = Fake/' >data/conf/openqa.ini

docker network create openQA

docker run -d --network openQA -e POSTGRES_PASSWORD=openqa -e POSTGRES_USER=openqa -e POSTGRES_DB=openqa --net-alias=db --name openqa_db postgres
docker logs -f openqa_db

openssl req -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -subj '/CN=www.mydom.com/O=My Company Name LTD./C=DE' -out data/certs/ssl.crt/server.crt -keyout data/certs/ssl.key/server.key
cp data/certs/ssl.crt/server.crt data/certs/ssl.crt/ca.crt

volumes="-v $(pwd)/data:/data"
certificates="-v $(pwd)/data/certs/ssl.crt:/etc/apache2/ssl.crt -v $(pwd)/data/certs/ssl.key:/etc/apache2/ssl.key"

docker build openQA/container/webui -t openqa_webui
docker run -d --network openQA $volumes $certificates -p 80:80 --net-alias=openqa_webui --name openqa_webui openqa_webui
docker logs -f openqa_webui

To have videos is great but we should not encourage users to use the development repos. Also our goal should be "single-command" solutions. As openqa-bootstrap already supports to take jobs to clone as parameters I suggest to start with that

This seems very similar to what openqa-bootstrap-container does except that it uses docker as a runtime. Maybe we want that as a re-usable script?

The video teaches the user how to build and deploy openQA, and that's what this ticket asks for, so I like it. Not sure if this is exactly what I was expecting since the audience wasn't defined. Deplying a container probably makes it very widely accessible so I like that. My main concern would be the use of the git repo as already mentioned by @okurz... thanks to #43718 we presumably have versioned images that can be used here and avoid git?

cdywan wrote:

This seems very similar to what openqa-bootstrap-container does except that it uses docker as a runtime. Maybe we want that as a re-usable script?

Yes, that was mentioned as the second point within the suggestions.

The video teaches the user how to build and deploy openQA, and that's what this ticket asks for, so I like it. Not sure if this is exactly what I was expecting since the audience wasn't defined.

Well, the user story has an actor "potential user of openQA" and the goal is "to be hooked" :)

And yes, a container based solution was suggested. That is fine. But as we already have an approach as documented in http://open.qa/docs/#bootstrapping that shows how an openQA installation can be installed and immediately start a clone-job I consider the multiple steps shown in the video as a step backwards.

okurz wrote:

Well, the user story has an actor "potential user of openQA" and the goal is "to be hooked" :)

And yes, a container based solution was suggested. That is fine. But as we already have an approach as documented in http://open.qa/docs/#bootstrapping that shows how an openQA installation can be installed and immediately start a clone-job I consider the multiple steps shown in the video as a step backwards.

Those instructions say, here's a command, go read a file to figure out how to actually use it. I think that's what we want to improve here.

1. What concrete steps are missing here in the typical case?
2. Is it desirable to customize the container like in #76978#note-7
3. Certificate setup like in #76978#note-7
4. Running an actual test, not covered yet

Might be fun to piggyback on k3s for this. https://k3s.io Then the user would need literally nothing but a Linux kernel and Bash to get started with openQA. You might not like curl | bash one-liners for one reason or another, but that's what gets people hooked.

exactly :) That makes sense.

@ilausuch Take a look at openqa-bootstrap{,-container for inspiration

• Avoid git repos - keep in mind the new user who just wants to run a test
• Pretend there's nothing installed, don't assume Tumbleweed/SLE (we cover that use case with existing scripts, no need to replicate that)
• Use --device=/dev/kvm
• Keep it simple, one or two lines if possible
• Make sure any setup of e.g. qemu/isotovideo is inside the container

Trying to figure out how to run the worker without being root

The group of the kvm is 128 that doesn't belongs to any group in the container

4a8e60738367:/ # ls -lh /dev/kvm 
crw-rw---- 1 root 128 10, 232 Feb 11 08:34 /dev/kvm

I tried to assign the group using the number, but doesn't work

usermod -g 128 _openqa-worker
usermod: group '128' does not exist

thanks George we found a solution using groupmod

groupmod -g 128 kvm

cdywan wrote:

@ilausuch Take a look at openqa-bootstrap{,-container for inspiration

• Avoid git repos - keep in mind the new user who just wants to run a test

I have this in mind, the previous recordings were tests for the final recording that will be using the container repository

• Pretend there's nothing installed, don't assume Tumbleweed/SLE (we cover that use case with existing scripts, no need to replicate that)

Right now, these steps are valid for in a Ubuntu too. So we could consider that is independent to the linux distro. Only, the user has to install docker and as far as I know kvm too

• Use --device=/dev/kvm

Yes, the PR associated to this point is this https://github.com/os-autoinst/openQA/pull/3715

• Keep it simple, one or two lines if possible

There are several steps we cannot avoid, because we have different containers, in this case a progress, webUI and worker has to be deployed.

Additionally some aspects like volumes and certificates has to be provided from the user.

• Volumes: We need to set-up an initial directory structure and provide to the containers via volumes. For instance, the configuration on webui and worker container doesn't exists. Has to be provided.
• Certificates: In some previous discussions we agree that the certificates has to be provided by the user.
• Tests: If we are working a generic tool the tests and needles cannot be part of the worker container image because we don't know what tests the user is going to use. Then has to be downloaded and put in the right directory by the user

Update: I am preparing a simple script to help the user to run all of these in two or there lines

• Make sure any setup of e.g. qemu/isotovideo is inside the container

That works. The video of the job is stored on assets can can be played.

ilausuch wrote:

• Keep it simple, one or two lines if possible

There are several steps we cannot avoid, because we have different containers, in this case a progress, webUI and worker has to be deployed.

Additionally some aspects like volumes and certificates has to be provided from the user.

• Volumes: We need to set-up an initial directory structure and provide to the containers via volumes. For instance, the configuration on webui and worker container doesn't exists. Has to be provided.
• Certificates: In some previous discussions we agree that the certificates has to be provided by the user.
• Tests: If we are working a generic tool the tests and needles cannot be part of the worker container image because we don't know what tests the user is going to use. Then has to be downloaded and put in the right directory by the user

Update: I am preparing a simple script to help the user to run all of these in two or there lines

I commented on the PR but for the record also replying here.

Do we really want a whole CLI here? Can't we use docker-compose for the setup of volumes, different containers etc.?

I could see if letting users supply certificates or openSUSE test repos is not trivial, but then that's the only things the script should be doing.

Working on the docker-compose file
Some considerations:

• Because all parts work in different containers, the OPENQA_SCHEDULER_HOST and OPENQA_WEB_SOCKETS_HOST env variables need to be set in the webui containers and the scheduler so that when they communicate properly
• The directory factory needs the subdirectories: iso,hdd,other,tmp
• The directories factory and subdirectories, and testresults has to be read-write
• To add a worker in the docker-compose a API key/secret is needed from the beginning
• The scheduler, webui, and others tries to create the tables on the database at the same time creating failures. Is needed to delay some of them to prevent this.

Preparing the PR. Now It can execute jobs, but doesn't show anything in the live view.

ilausuch wrote:

• The directory factory needs the subdirectories: iso,hdd,other,tmp
• The directories factory and subdirectories, and testresults has to be read-write

Could we have those created on demand by the worker? It occurs to me this is something I'd also do as part of a development setup as well, ensure e.g. share/factory/hdd exists because otherwise cloning won't work. Just empty folders are expected here. 🤔

cdywan wrote:

ilausuch wrote:

• The directory factory needs the subdirectories: iso,hdd,other,tmp
• The directories factory and subdirectories, and testresults has to be read-write

Could we have those created on demand by the worker? It occurs to me this is something I'd also do as part of a development setup as well, ensure e.g. share/factory/hdd exists because otherwise cloning won't work. Just empty folders are expected here. 🤔

This was my first approach, but it's more easy than that. The docker-compose can do it. I've just checked that work

Created the PR that uses the docker-container to deploy the entire openQA system
https://github.com/os-autoinst/openQA/pull/3755

Considerations:

• The user must install the tests on workdir/tests
• To run opensuse and other SUSE tests execute: docker exec -t openqa_worker zypper in -y os-autoinst-distri-opensuse-deps

A very simple way for cloning a custom job from other openQA instance can be just something like:

podman run --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -it registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman exec openqa /usr/share/openqa/script/fetchneedles
podman exec openqa openqa-clone-job https://openqa.opensuse.org/tests/4109159
podman exec openqa openqa-cli monitor 1
podman exec openqa openqa-cli api --pretty jobs/1

The full "run one job locally" equivalent from o3/osd could be made even simpler as we have some built-in support. The full workflow can be summed up into:

podman run --detach --env skip_suse_tests="" --env skip_suse_specifics="" --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -ti registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman wait --condition healthy openqa
podman exec openqa openqa-clone-job https://openqa.opensuse.org/tests/4109159
podman exec openqa openqa-cli monitor 1
podman exec openqa openqa-cli api --pretty jobs/1
podman stop openqa

You can of course do something more "wild" and schedule a custom job as we did as a part of openqa-in-openqa test:

podman run --detach --env skip_suse_tests="" --env skip_suse_specifics="" --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -ti registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman wait --condition healthy openqa
wget -O scenario-definitions.yaml https://raw.githubusercontent.com/os-autoinst/os-autoinst-distri-example/main/scenario-definitions.yaml
podman cp scenario-definitions.yaml openqa:scenario-definitions.yaml
podman exec -ti openqa openqa-cli schedule --monitor --param-file SCENARIO_DEFINITIONS_YAML=scenario-definitions.yaml DISTRI=example VERSION=0 FLAVOR=DVD ARCH=x86_64 TEST=simple_boot _GROUP_ID=0 BUILD=test CASEDIR=https://github.com/os-autoinst/os-autoinst-distri-example.git NEEDLES_DIR=%%CASEDIR%%/needles
podman exec openqa openqa-cli api jobs/1 | jq -r .job.result
podman stop openqa

I will enhance the documentation to have those examples and create a short "video" featuring this.