action #76978
closedcoordination #127031: [saga][epic] openQA for SUSE customers
coordination #80150: [epic] Scale out openQA: Easier openQA setup
How to run an openQA test in 5 minutes size:M
Description
User story¶
As a potential user of openQA I want to run an openQA test in 5 minutes to be hooked
Acceptance criteria¶
- AC1: An openQA instance can be deployed and is used to execute a new (empty) test within 5 minutes (not counting download times on slow connections)
- AC2: Documentation exists for how to do it
Suggestions¶
- Start with existing container solution (and follow the docs)
- Follow-up with what has been done in #129883
- Have a workflow (or test or something) that uses the quick bootstrap or comparable approach and runs tests based on github.com/os-autoinst/os-autoinst-distri-example
- record a video and make some noiz about it :) Either a real life video with explanation how you use it or also something like https://asciinema.org/
- maybe create/adapt some tool like
osc
but for openqa that will allow users to run something likeosc build
Further details¶
Inspired by talking to bzoltan :)
Files
Updated by okurz about 4 years ago
https://github.com/os-autoinst/openQA/pull/3511 as a related first step.
EDIT: merged. Next steps as outlined in suggestions
Updated by okurz about 4 years ago
- Description updated (diff)
- Status changed from New to Workable
- Target version changed from future to Ready
- Parent task set to #80142
Updated by openqa_review almost 4 years ago
- Due date set to 2021-02-13
Setting due date based on mean cycle time of SUSE QE Tools
Updated by ilausuch almost 4 years ago
I recorded a video to show how to deploy openqa_webui container
https://asciinema.org/a/x8UYkEysNsGlAfjCamvzLmjh0
git clone https://github.com/os-autoinst/openQA.git
mkdir -p data/factory/{iso,hdd,other,tmp} data/{testresults,tests,conf} data/certs/{ssl.crt,ssl.key}
cp openQA/container/openqa_data/data.template/conf/* data/conf
cp openQA/container/webui/conf/database.ini data/conf/
cat /tmp/openqa.ini | sed 's/method = OpenID/method = Fake/' >data/conf/openqa.ini
docker network create openQA
docker run -d --network openQA -e POSTGRES_PASSWORD=openqa -e POSTGRES_USER=openqa -e POSTGRES_DB=openqa --net-alias=db --name openqa_db postgres
docker logs -f openqa_db
openssl req -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -subj '/CN=www.mydom.com/O=My Company Name LTD./C=DE' -out data/certs/ssl.crt/server.crt -keyout data/certs/ssl.key/server.key
cp data/certs/ssl.crt/server.crt data/certs/ssl.crt/ca.crt
volumes="-v $(pwd)/data:/data"
certificates="-v $(pwd)/data/certs/ssl.crt:/etc/apache2/ssl.crt -v $(pwd)/data/certs/ssl.key:/etc/apache2/ssl.key"
docker build openQA/container/webui -t openqa_webui
docker run -d --network openQA $volumes $certificates -p 80:80 --net-alias=openqa_webui --name openqa_webui openqa_webui
docker logs -f openqa_webui
Updated by okurz almost 4 years ago
To have videos is great but we should not encourage users to use the development repos. Also our goal should be "single-command" solutions. As openqa-bootstrap already supports to take jobs to clone as parameters I suggest to start with that
Updated by livdywan almost 4 years ago
This seems very similar to what openqa-bootstrap-container
does except that it uses docker as a runtime. Maybe we want that as a re-usable script?
The video teaches the user how to build and deploy openQA, and that's what this ticket asks for, so I like it. Not sure if this is exactly what I was expecting since the audience wasn't defined. Deplying a container probably makes it very widely accessible so I like that. My main concern would be the use of the git repo as already mentioned by @okurz... thanks to #43718 we presumably have versioned images that can be used here and avoid git?
Updated by livdywan almost 4 years ago
- Status changed from Workable to In Progress
Updated by okurz almost 4 years ago
cdywan wrote:
This seems very similar to what
openqa-bootstrap-container
does except that it uses docker as a runtime. Maybe we want that as a re-usable script?
Yes, that was mentioned as the second point within the suggestions.
The video teaches the user how to build and deploy openQA, and that's what this ticket asks for, so I like it. Not sure if this is exactly what I was expecting since the audience wasn't defined.
Well, the user story has an actor "potential user of openQA" and the goal is "to be hooked" :)
And yes, a container based solution was suggested. That is fine. But as we already have an approach as documented in http://open.qa/docs/#bootstrapping that shows how an openQA installation can be installed and immediately start a clone-job I consider the multiple steps shown in the video as a step backwards.
Updated by livdywan almost 4 years ago
okurz wrote:
Well, the user story has an actor "potential user of openQA" and the goal is "to be hooked" :)
And yes, a container based solution was suggested. That is fine. But as we already have an approach as documented in http://open.qa/docs/#bootstrapping that shows how an openQA installation can be installed and immediately start a clone-job I consider the multiple steps shown in the video as a step backwards.
Those instructions say, here's a command, go read a file to figure out how to actually use it. I think that's what we want to improve here.
- What concrete steps are missing here in the typical case?
- Is it desirable to customize the container like in #76978#note-7
- Certificate setup like in #76978#note-7
- Running an actual test, not covered yet
Updated by kraih almost 4 years ago
Might be fun to piggyback on k3s for this. https://k3s.io Then the user would need literally nothing but a Linux kernel and Bash to get started with openQA. You might not like curl | bash
one-liners for one reason or another, but that's what gets people hooked.
Updated by livdywan almost 4 years ago
@ilausuch Take a look at openqa-bootstrap{,-container
for inspiration
- Avoid git repos - keep in mind the new user who just wants to run a test
- Pretend there's nothing installed, don't assume Tumbleweed/SLE (we cover that use case with existing scripts, no need to replicate that)
- Use
--device=/dev/kvm
- Keep it simple, one or two lines if possible
- Make sure any setup of e.g. qemu/isotovideo is inside the container
Updated by ilausuch almost 4 years ago
Thanks for all your help.
I recorded a video showing how to setup the worker https://asciinema.org/a/390320
The script to have the worker running is this:
$ chmod a+w data/testresults
In this section we are going to use only opensuse test suites, but if you want more add them following the example:
$ git clone git@github.com:os-autoinst/os-autoinst-distri-opensuse.git
$ ln -s os-autoinst-distri-opensuse opensuse
$ git clone https://github.com/os-autoinst/os-autoinst-needles-opensuse.git
$ mv os-autoinst-needles-opensuse/ opensuse/products/opensuse/needles
Create a key/secret pair on your webUI and replace the example ones to configure the client.conf
$ echo -e “[openqa_webui]\nkey = 5AD3533BB586FD54\nsecret = AE0A2A27FD82D530” > data/conf/client.conf
Ensure that the variable volumes is set
$ volumes=”-v $(pwd)/data:/data”
Build the worker
$ docker build openQA/container/worker -t openqa_worker
Run the worker
$ docker run -d —network openQA $volumes —device=/dev/kvm —privileged —name openqa_worker openqa_worker
You can launch a test accessing to the web UI
$ docker exec -ti openqa_webui /usr/share/openqa/script/clone_job.pl —apikey 5AD3533BB586FD54 —apisecret AE0A2A27FD82D530 https://openqa.opensuse.org/tests/1620808
Updated by ilausuch almost 4 years ago
Trying to figure out how to run the worker without being root
The group of the kvm is 128 that doesn't belongs to any group in the container
4a8e60738367:/ # ls -lh /dev/kvm
crw-rw---- 1 root 128 10, 232 Feb 11 08:34 /dev/kvm
I tried to assign the group using the number, but doesn't work
usermod -g 128 _openqa-worker
usermod: group '128' does not exist
Updated by ilausuch almost 4 years ago
thanks George we found a solution using groupmod
groupmod -g 128 kvm
Updated by ilausuch almost 4 years ago
cdywan wrote:
@ilausuch Take a look at
openqa-bootstrap{,-container
for inspiration
- Avoid git repos - keep in mind the new user who just wants to run a test
I have this in mind, the previous recordings were tests for the final recording that will be using the container repository
- Pretend there's nothing installed, don't assume Tumbleweed/SLE (we cover that use case with existing scripts, no need to replicate that)
Right now, these steps are valid for in a Ubuntu too. So we could consider that is independent to the linux distro. Only, the user has to install docker and as far as I know kvm too
- Use
--device=/dev/kvm
Yes, the PR associated to this point is this https://github.com/os-autoinst/openQA/pull/3715
- Keep it simple, one or two lines if possible
There are several steps we cannot avoid, because we have different containers, in this case a progress, webUI and worker has to be deployed.
Additionally some aspects like volumes and certificates has to be provided from the user.
- Volumes: We need to set-up an initial directory structure and provide to the containers via volumes. For instance, the configuration on webui and worker container doesn't exists. Has to be provided.
- Certificates: In some previous discussions we agree that the certificates has to be provided by the user.
- Tests: If we are working a generic tool the tests and needles cannot be part of the worker container image because we don't know what tests the user is going to use. Then has to be downloaded and put in the right directory by the user
Update: I am preparing a simple script to help the user to run all of these in two or there lines
- Make sure any setup of e.g. qemu/isotovideo is inside the container
That works. The video of the job is stored on assets can can be played.
Updated by livdywan almost 4 years ago
- Due date changed from 2021-02-13 to 2021-02-19
Bumping the Due date since enabling kvm isn't fully solved yet and there were open questions with scripting installation. See the record of the last conversation above.
Updated by livdywan almost 4 years ago
- Due date changed from 2021-02-19 to 2021-02-26
Update: I am preparing a simple script to help the user to run all of these in two or there lines
Updated by livdywan almost 4 years ago
ilausuch wrote:
- Keep it simple, one or two lines if possible
There are several steps we cannot avoid, because we have different containers, in this case a progress, webUI and worker has to be deployed.
Additionally some aspects like volumes and certificates has to be provided from the user.
- Volumes: We need to set-up an initial directory structure and provide to the containers via volumes. For instance, the configuration on webui and worker container doesn't exists. Has to be provided.
- Certificates: In some previous discussions we agree that the certificates has to be provided by the user.
- Tests: If we are working a generic tool the tests and needles cannot be part of the worker container image because we don't know what tests the user is going to use. Then has to be downloaded and put in the right directory by the user
Update: I am preparing a simple script to help the user to run all of these in two or there lines
I commented on the PR but for the record also replying here.
Do we really want a whole CLI here? Can't we use docker-compose
for the setup of volumes, different containers etc.?
I could see if letting users supply certificates or openSUSE test repos is not trivial, but then that's the only things the script should be doing.
Updated by ilausuch almost 4 years ago
Working on the docker-compose file
Some considerations:
- Because all parts work in different containers, the OPENQA_SCHEDULER_HOST and OPENQA_WEB_SOCKETS_HOST env variables need to be set in the webui containers and the scheduler so that when they communicate properly
- The directory factory needs the subdirectories: iso,hdd,other,tmp
- The directories factory and subdirectories, and testresults has to be read-write
- To add a worker in the docker-compose a API key/secret is needed from the beginning
- The scheduler, webui, and others tries to create the tables on the database at the same time creating failures. Is needed to delay some of them to prevent this.
Preparing the PR. Now It can execute jobs, but doesn't show anything in the live view.
Updated by livdywan almost 4 years ago
ilausuch wrote:
- The directory factory needs the subdirectories: iso,hdd,other,tmp
- The directories factory and subdirectories, and testresults has to be read-write
Could we have those created on demand by the worker? It occurs to me this is something I'd also do as part of a development setup as well, ensure e.g. share/factory/hdd
exists because otherwise cloning won't work. Just empty folders are expected here. 🤔
Updated by ilausuch almost 4 years ago
- Due date changed from 2021-02-26 to 2021-03-05
Updated by ilausuch almost 4 years ago
cdywan wrote:
ilausuch wrote:
- The directory factory needs the subdirectories: iso,hdd,other,tmp
- The directories factory and subdirectories, and testresults has to be read-write
Could we have those created on demand by the worker? It occurs to me this is something I'd also do as part of a development setup as well, ensure e.g.
share/factory/hdd
exists because otherwise cloning won't work. Just empty folders are expected here. 🤔
This was my first approach, but it's more easy than that. The docker-compose can do it. I've just checked that work
Updated by ilausuch almost 4 years ago
Created the PR that uses the docker-container to deploy the entire openQA system
https://github.com/os-autoinst/openQA/pull/3755
Considerations:
- The user must install the tests on workdir/tests
- To run opensuse and other SUSE tests execute: docker exec -t openqa_worker zypper in -y os-autoinst-distri-opensuse-deps
Updated by livdywan over 3 years ago
- Blocked by action #89719: docker-compose up fails on master added
Updated by okurz over 3 years ago
- Status changed from In Progress to Blocked
We discussed in the daily 2021-03-09 that we encountered multiple issues with the current docker-compose.yaml file so that the current docker-compose.yaml which we should cover in individual tickets so that we can progress within cycle times. Also we need an automatic check for docker-compose, see #89722
Updated by livdywan over 3 years ago
- Blocked by action #89731: containers: The deploy using docker-compose is not stable and eventually fails added
Updated by livdywan over 3 years ago
- Blocked by deleted (action #89731: containers: The deploy using docker-compose is not stable and eventually fails )
Updated by okurz over 3 years ago
- Blocked by action #88451: Static validation for containers added
Updated by livdywan over 3 years ago
- Related to action #89752: containers: Add a worker service as part of the docker-compose added
Updated by livdywan over 3 years ago
- Related to action #89755: container: Fix missing shared directories and its permissions added
Updated by ilausuch over 3 years ago
- Status changed from Blocked to In Progress
I unblocked because recent related tickets are closed and with what we have now it could be possible. I am going to explore if is it possible now run a full env with the web UI using the docker-compose and worker using a few commands. If is not possible I'll write down what are the reasons and then create new tickets if are necessary
Updated by ilausuch over 3 years ago
- Status changed from In Progress to Blocked
During this new attempt to run a job I found a problem:
webui_2 | [warn] Unable to wakeup scheduler: Connection refused
I tested the access to the scheduler from this webui and works
$ ping scheduler
PING scheduler (192.168.80.8) 56(84) bytes of data.
64 bytes from webui_scheduler_1.webui_default (192.168.80.8): icmp_seq=1 ttl=64 time=0.126 ms
$ telnet scheduler 9529
Trying 192.168.80.8...
Connected to scheduler.
Escape character is '^]'.
So, maybe is a configuration problem or other env variables needed
To prepare the environment:
$ cd openQA/containers/webui
$ docker-compose build
# Edit openqa.ini and set auth method to Fake
$ docker-compose up -d
# Now you can access to the UI at http://localhost:9526 and login. If you want you can create a new credentials, there are ones by default
# go to the directory workdir/data/tests and download the test suites you want, for instance opensuse
$ sudo git clone https://github.com/os-autoinst/os-autoinst-distri-opensuse.git
$ sudo mv os-autoinst-distri-opensuse opensuse
$ sudo git clone https://github.com/os-autoinst/os-autoinst-needles-opensuse.git
$ sudo mv os-autoinst-needles-opensuse/ opensuse/products/opensuse/needles
# Move to the worker directory
$ ln -s ../webui/workdir/
# Edit conf/worker.ini and change HOST to http://nginx:9526
# Edit conf/client.conf and change [openqa_webui] to [nginx]
$ docker run -d --network=webui_default -v $(pwd)/workdir/data/factory:/data/factory:rw -v $(pwd)/workdir/data/tests:/data/tests:ro -v $(pwd)/conf:/data/conf:ro --device=/dev/kvm --privileged openqa_worker --name openqa_worker
# Check that the worker has connected
$ docker logs -f openqa_worker
To launch the test:
$ docker-compose exec webui bash
$ /usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host http://nginx:9526 1738950
# This download all the assets correctly
I created a ticket to solve this problem #92833 and this one will be blocked until the other is solved
Updated by ilausuch over 3 years ago
- Related to action #92833: containers: Web UI cannot connect to scheduler added
Updated by ilausuch over 3 years ago
- Blocked by action #92893: containers, docker-compose: Ensure that the scheduler can connect to the websockets container size:M added
Updated by okurz over 3 years ago
- Related to coordination #81060: [epic] openQA web UI in kubernetes added
Updated by okurz about 3 years ago
- Status changed from Blocked to New
- Assignee deleted (
okurz)
blocker #92893 resolved.
Updated by okurz over 1 year ago
- Copied to action #129883: all-in-one openQA container solution added
Updated by livdywan over 1 year ago
- Status changed from New to Blocked
- Assignee set to okurz
Blocking on #129883
Updated by okurz over 1 year ago
- Description updated (diff)
- Status changed from Blocked to New
- Assignee deleted (
okurz)
Updated by okurz over 1 year ago
- Subject changed from How to run an openQA test in 5 minutes to How to run an openQA test in 5 minutes size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by tinita over 1 year ago
- Target version changed from Ready to future
Setting to future for now because our backlog is full
Updated by openqa_review 7 months ago
- Due date set to 2024-05-10
Setting due date based on mean cycle time of SUSE QE Tools
Updated by jbaier_cz 7 months ago
A very simple way for cloning a custom job from other openQA instance can be just something like:
podman run --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -it registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman exec openqa /usr/share/openqa/script/fetchneedles
podman exec openqa openqa-clone-job https://openqa.opensuse.org/tests/4109159
podman exec openqa openqa-cli monitor 1
podman exec openqa openqa-cli api --pretty jobs/1
The full "run one job locally" equivalent from o3/osd could be made even simpler as we have some built-in support. The full workflow can be summed up into:
podman run --detach --env skip_suse_tests="" --env skip_suse_specifics="" --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -ti registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman wait --condition healthy openqa
podman exec openqa openqa-clone-job https://openqa.opensuse.org/tests/4109159
podman exec openqa openqa-cli monitor 1
podman exec openqa openqa-cli api --pretty jobs/1
podman stop openqa
You can of course do something more "wild" and schedule a custom job as we did as a part of openqa-in-openqa test:
podman run --detach --env skip_suse_tests="" --env skip_suse_specifics="" --name openqa --device /dev/kvm -p 1080:80 -p 1443:443 --rm -ti registry.opensuse.org/devel/openqa/containers/openqa-single-instance
podman wait --condition healthy openqa
wget -O scenario-definitions.yaml https://raw.githubusercontent.com/os-autoinst/os-autoinst-distri-example/main/scenario-definitions.yaml
podman cp scenario-definitions.yaml openqa:scenario-definitions.yaml
podman exec -ti openqa openqa-cli schedule --monitor --param-file SCENARIO_DEFINITIONS_YAML=scenario-definitions.yaml DISTRI=example VERSION=0 FLAVOR=DVD ARCH=x86_64 TEST=simple_boot _GROUP_ID=0 BUILD=test CASEDIR=https://github.com/os-autoinst/os-autoinst-distri-example.git NEEDLES_DIR=%%CASEDIR%%/needles
podman exec openqa openqa-cli api jobs/1 | jq -r .job.result
podman stop openqa
I will enhance the documentation to have those examples and create a short "video" featuring this.
Updated by jbaier_cz 7 months ago
- File openqa-dark.gif openqa-dark.gif added
- File openqa-light.gif openqa-light.gif added
Not sure if we like light or dark theme more, but this could be the first proof-of-concept. As a next step I will prepare some PR into our documentation.
Updated by jbaier_cz 7 months ago
- Related to action #155491: Extend documentation for single-instance-container covering at least how to trigger/clone existing jobs size:S added
Updated by jbaier_cz 7 months ago
With #159747 almost out of the way, I created https://github.com/os-autoinst/openQA/pull/5611 for including a short demo inside our documentation. There is also an automated way to regenerate the image if needed and to create more if wanted. This might also satisfy the conditions for #155491. If needed, I can also include the information from the recording in a written form inside the same documentation section.
Updated by jbaier_cz 7 months ago
- Due date deleted (
2024-05-10) - Status changed from Blocked to Resolved
Documentation updated in https://github.com/os-autoinst/openQA/pull/5625