Project

General

Profile

action #73450

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

POC: Create openQA worker container image (feature)

Added by plorinc 10 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-10-16
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

User story

openQA worker is preinstalled with all it's dependencies (os-autoinst, etc.)
The worker has access to the test database
The worker can start and evaluate tests orchestrated by the openQA Web App

The worker can run tests for a simple package (e.g. vi)

Acceptance criteria

  • AC1: Container started based on this image is able to run on the Docker engine
  • AC2: Full Green Stack: All software components are available in the (open)SUSE repositories
  • AC3: An automated way is provided to fill the database with necessary tables and schemas, and a predefined set of test data
  • AC4: Non-static data is stored on container volumes, if possible
  • AC5: Deliverable is an image stored in a registry, a Dockerfile (with instructions on how to create a container from the image) or a Compose file

Tasks

Further details


Related issues

Related to openQA Project - action #73447: POC: Create openQA Web Application container image (feature)Resolved2020-10-16

Related to openQA Project - coordination #43706: [epic] Generate "download&use" docker image of openQA for SUSE QABlocked2018-11-13

Related to openQA Project - action #43715: Update upstream dockerfiles to provide an easy to use docker image of workersResolved2018-11-13

History

#1 Updated by okurz 9 months ago

  • Description updated (diff)
  • Due date set to 2020-10-31
  • Status changed from New to Feedback
  • Assignee set to okurz
  • Target version set to Ready

In principle I see this already fulfilled by https://github.com/os-autoinst/openQA/blob/master/docker/worker/Dockerfile but I want to clarify that your request is completely covered.

  • I assume when you say "SUSE repositories" that "openSUSE" is ok?
  • Can you clarify on "AC3: An automated way is provided to fill the database with necessary tables and schemas, and a predefined set of test data": What do you want to achieve with that and what do you mean with "predefined set of test data" in particular?
  • Can you clarify on "AC4: Non-static data is stored on container volumes, if possible": Do you mean something like an external temporary directory passed into the container as the worker's cache and/or pool directory?

#2 Updated by okurz 9 months ago

  • Related to action #73447: POC: Create openQA Web Application container image (feature) added

#3 Updated by okurz 9 months ago

  • Related to coordination #43706: [epic] Generate "download&use" docker image of openQA for SUSE QA added

#4 Updated by okurz 9 months ago

  • Related to action #43715: Update upstream dockerfiles to provide an easy to use docker image of workers added

#5 Updated by ilausuch 9 months ago

This case is similar to my comment on #73447 and a PR that is currently WIP PR https://github.com/os-autoinst/openQA/pull/3475 that covers some ACs

#6 Updated by cdywan 9 months ago

plorinc wrote:

Acceptance criteria

  • AC1: Container started based on this image is able to run on the Docker engine

This is covered by #43715 already.

  • AC5: Deliverable is an image stored in a registry, a Dockerfile (with instructions on how to create a container from the image) or a Compose file

This is covered by #43718 already.

#7 Updated by cdywan 9 months ago

plorinc wrote:

Acceptance criteria

  • AC2: Full Green Stack: All software components are available in the (open)SUSE repositories

What (open)SUSE repositories are we talking about here?

The images are based on Leap 15.1 (worker and webui respectively), and use these repos on top:

zypper ar -f obs://devel:openQA/openSUSE_Leap_15.1 openQA
zypper ar -f obs://devel:openQA:Leap:15.1/openSUSE_Leap_15.1 openQA-perl-modules

#8 Updated by okurz 9 months ago

cdywan is that you trying to answer the question to plorinc or are you just adding additional information for the current state?

#9 Updated by okurz 9 months ago

  • Due date deleted (2020-10-31)
  • Status changed from Feedback to New
  • Assignee deleted (okurz)
  • Priority changed from Normal to Low
  • Target version changed from Ready to future

plorinc can you please look into the questions from #73450#note-1

#10 Updated by cdywan 9 months ago

Can we regard this as a duplicate of #43715 and #43718?

#11 Updated by okurz 8 months ago

Well, I prefer to not regard this as duplicate as the specific stakeholder expectations might be different. But this is what I hoped I could learn from plorinc when he has the possibility to answer.

#12 Updated by okurz 8 months ago

  • Parent task set to #80142

#13 Updated by plorinc 8 months ago

okurz wrote:

In principle I see this already fulfilled by https://github.com/os-autoinst/openQA/blob/master/docker/worker/Dockerfile but I want to clarify that your request is completely covered.

The Dockerfile looks good. Do you have any guide on how to start a container from the image?

  • I assume when you say "SUSE repositories" that "openSUSE" is ok?

Leap 15.2 is perfectly ok.

  • Can you clarify on "AC3: An automated way is provided to fill the database with necessary tables and schemas, and a predefined set of test data": What do you want to achieve with that and what do you mean with "predefined set of test data" in particular?

The image will also be used in test/training scenarios completely isolated from the production system.
In these use cases we needed to put the system into a predefined state, for example by filling the database with a snapshot or a prepared dataset.

  • Can you clarify on "AC4: Non-static data is stored on container volumes, if possible": Do you mean something like an external temporary directory passed into the container as the worker's cache and/or pool directory?

Persistent data should be stored on mounted docker volumes so if the container gets restarted (e.g. because of redeployment or software upgrade) the data won't lost.
Basically: configuration is stored inside the container, state is stored outside.

#14 Updated by plorinc 8 months ago

okurz wrote:

Well, I prefer to not regard this as duplicate as the specific stakeholder expectations might be different. But this is what I hoped I could learn from plorinc when he has the possibility to answer.

We don't plan to build the images with OBS so please don't regard this as a duplicate.

#15 Updated by okurz 8 months ago

  • Status changed from New to Resolved
  • Assignee set to ilausuch
  • Target version changed from future to Ready

plorinc wrote:

okurz wrote:

In principle I see this already fulfilled by https://github.com/os-autoinst/openQA/blob/master/docker/worker/Dockerfile but I want to clarify that your request is completely covered.

The Dockerfile looks good. Do you have any guide on how to start a container from the image?

Yes, see https://github.com/os-autoinst/openQA/blob/master/docs/Installing.asciidoc#container-based-setup or simply:

podman run --rm -it registry.opensuse.org/devel/openqa/containers15.2/openqa_worker:latest
  • Can you clarify on "AC3: An automated way is provided to fill the database with necessary tables and schemas, and a predefined set of test data": What do you want to achieve with that and what do you mean with "predefined set of test data" in particular?

The image will also be used in test/training scenarios completely isolated from the production system.
In these use cases we needed to put the system into a predefined state, for example by filling the database with a snapshot or a prepared dataset.

I have commented to a similar question regarding the webUI in #73447. The worker only needs simple command line parameters or configuration files which can be linked into the container.

  • Can you clarify on "AC4: Non-static data is stored on container volumes, if possible": Do you mean something like an external temporary directory passed into the container as the worker's cache and/or pool directory?

Persistent data should be stored on mounted docker volumes so if the container gets restarted (e.g. because of redeployment or software upgrade) the data won't lost.
Basically: configuration is stored inside the container, state is stored outside.

The worker forwards all relevant output data to the web UI. The only relevant data for the worker is the configuration mentioned above. So I don't see this as problem.

Let's see if we can cover the ACs:

  • AC1: Container started based on this image is able to run on the Docker engine

DONE: We tested with both "docker" and "podman". https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3Ddevel%3AopenQA lists the containers along with example commands based on podman as well as docker.

  • AC2: Full Green Stack: All software components are available in the (open)SUSE repositories

DONE: Available in openSUSE Tumbleweed as well as build.opensuse.org/project/show/devel:openQA

  • AC3: An automated way is provided to fill the database with necessary tables and schemas, and a predefined set of test data

DONE: N/A for the openQA worker

  • AC4: Non-static data is stored on container volumes, if possible

DONE: N/A for the openQA worker

  • AC5: Deliverable is an image stored in a registry, a Dockerfile (with instructions on how to create a container from the image) or a Compose file

DONE: All of that with https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3Ddevel%3AopenQA and https://github.com/os-autoinst/openQA/tree/master/docker

Same as in #73447 setting ilausuch as he was the main driver in the latest work, I am just collecting facts as result :)

Also available in: Atom PDF