QA (public) » openQA Project (public)

openqa-staging-2.qa.suse.de could be used for that - we discussed briefly after the daily that it's in need of an upgrade and could also be setup anew for k8s since we don't currently need two machines with the same setup

After initial investigation, it turned out that the conversion from docker-compose to kubernetes/helm is pretty trivial. By using kompose convert or kompose convert --chart, one can easily create kubernetes yaml files (or corresponding helm chart templates) which can be then deployed to kubernetes. Of course, the resulting files are only as good as the docker-compose file itself. In our case, it is more targeted to a development on the localhost than a production grade setup, so some tweaking in the file is still needed. Nevertheless, it still gives a nice file to start with. So basically with zypper in kompose helm kubernetes1.23-client one can get the proper tools to do the necessary bits.

sed -i -e 's#image: openqa_worker#image: registry.opensuse.org/devel/openqa/containers15.3/openqa_worker:latest#' docker-compose.yaml
export OPENQA_WORKER_REPLICAS=1
kompose convert --with-kompose-annotation=false --volumes emptyDir -c -o worker
helm install --generate-name worker

That will basically install the worker into the cluster. Of course, it will not start as the configuration is missing and there are no data. I covered the configuration by creating a ConfigMap. The data (factory and tests folders) can be solved in two different ways:

1. Creating a PersistentVolumeClaims with the proper content, either by having a separate container (similar to what openqa_data container provides) which will obtain the data somehow; by having a NFS mount, or some other option. In this way, the worker should also be able to install missing dependencies if the install_deps.* script is found. I will need to invest some time to explore this way more.

2. By enabling worker-cache, the worker can download assets/tests/needles itself. This needs rsync to be installed in the worker image (this is missing in the current Dockerfile) and before the main /run_openqa_worker.sh, two additional binaries needs to be started (with proper permissions): openqa-worker-cacheservice-minion and openqa-workercache-daemon. In this case, there is also no install_deps.sh, so the test dependencies will not be installed automatically. I can probably make this work by introducing another key in the ConfigMap and mounting that key directly into tests/install_deps.sh

After those manual modifications, I was able to successfully install a worker inside k3s cluster, register it against my personal openQA instance and clone a job: http://polaris.suse.cz/tests/2418; the next steps are create the necessary yaml files to automate the changes I made, quickly investigate some of the pitfalls (like non working CACHE_MIN_FREE_PERCENTAGE) and wrap this up in form of a very basic helm chart.

The item "Figure out if is necessary to publish the helm chart and where" could be simply answered by a howto page https://helm.sh/docs/howto/chart_releaser_action/ where a separate GitHub repository for charts and accompanied GitHub page for the Helm repository are suggested.

More has been added to the PR. We now can create a fully functioning openQA webui and worker: http://odin.qam.suse.cz/tests/3. The setup so far can be done only using the k3s ingress (so without nginx), the only small change needed is in https://github.com/os-autoinst/openQA/pull/4653.

The chart still misses some features, there are for example no simple way, how to get data inside the containers. The customization of the charts is also kept on minimal level (at least the basic variables provided by a helm sample template should be added in the near feature). Also, there is no tests yet.