QA » openQA Project

Let's consider this Blocked in the sense that the steps required are the same with a focus on the worker vs. the web UI.

Before to work on this ticket I would like to complete this one https://progress.opensuse.org/issues/69355 because it's related and maybe dependent on how this is resolved

One interesting thing should we consider is about the --link parameter in the docker run for the workers. There is an alert about that in this link https://docs.docker.com/network/links/

Two initial problems to fix during the build

• Package 'qemu-uefi-aarch64' not found
• /root/qemu/kvm-mknod.sh: line 6: gunzip: command not found

https://github.com/os-autoinst/openQA/pull/3475

New problems to solve

• gzip: /proc/config.gz: No such file or directory
• mknod: /dev/kvm: File exists
• Unable to make /dev/kvm node; software emulation will be used (This can happen if the container is run without -privileged)

In the last case, I think is because of configuration

Doing some tests on the worker I realized that the worker cannot start because:

[info] [pid:44] Project dir for host http://webui_haproxy_1 is /var/lib/openqa/share
[info] [pid:44] Registering with openQA http://webui_haproxy_1
[info] [pid:44] Establishing ws connection via ws://webui_haproxy_1/api/v1/ws/1
[warn] [pid:44] Unable to upgrade to ws connection via http://webui_haproxy_1/api/v1/ws/1 - trying again in 10 seconds

Facts:

• The worker can connect to the web UI API and authentificate the user
• The worker cannot connect to the websockets

To solve that Christian and me were working on a replacement for haproxy with nginx allowing the reverse proxy for the websokets. This seems to work

server {
  listen       80;
  listen       9526;
  server_name  localhost;

  location ~ /api/v1/ws/(.*) {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;

    rewrite ^//api/v1/ws/(.*)$ http://webui_websockets_1:9527/ws/$1;
    proxy_pass http://webui_websockets_1:9527;
    proxy_redirect off;
  }

  location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;

    proxy_pass http://webui_haproxy_1:9526;
    proxy_redirect off;
  }
}

Facts:

• nginx pass to the websockets the query
• nginx always return Not Found in plain text (tested from the worker, and also in the Christian environment)
• Christian checked that in O3 the websockets service also returns Not Found for the same path /api/v1/ws/1 however in O3 the worker is starting and in this test not. Why is the reason for that?

Ideas:

• check in a fake websockets service that nginx is sending the correct path
• Check what the worker spect from the websockets

Closely related with https://progress.opensuse.org/issues/69355. Both tickets are interdependent

I see that your NGINX config differs from what we have in our repository: https://github.com/os-autoinst/openQA/blob/master/etc/nginx/vhosts.d/openqa.conf

Maybe the lack of proxy_set_header Upgrade leads to the 404 response.

As mentioned in the chat yesterday: It is also possible to specify the web UI port directly within the worker config. The worker will then use this port +2 for the web sockets route making it unnecessary to proxy the web socket connection. Having NGINX proxying the web socket connection would be the nicer solution of course.

Works, preparing the https://progress.opensuse.org/issues/69355 finishing before

Thanks Marius, this worked and now we have a new PR for the web UI that solves all these problems. Now I am preparing this PR to use api keys and hosts for use with https://progress.opensuse.org/issues/69355

Additionally and following the subject of this task "provide an easy to use..." I created an script to launch a pool of workers
https://github.com/os-autoinst/openQA/pull/3495

I am preparing a PR to fix some problems in the Dockerfile
https://github.com/os-autoinst/openQA/pull/3523

In my home there are the packages I am working on
https://build.opensuse.org/package/show/home:ilausuch:branches:devel:openQA/openQA_container_image_worker_x86
https://build.opensuse.org/package/show/home:ilausuch:branches:devel:openQA/openQA_container_image_worker_aarch64

As well as I know, no tags are required for different architectures. There can be multiple builds of the same Dockerfile with the same tag for different architectures.

I created a PR to discuss about all of that
https://github.com/os-autoinst/openQA/pull/3533

Also there are two build on my home

• https://build.opensuse.org/package/show/home:ilausuch:branches:devel:openQA/openQA_container_image_worker_x86
• https://build.opensuse.org/package/show/home:ilausuch:branches:devel:openQA/openQA_container_image_worker_aarch64

The x86 works

Associated patch https://github.com/os-autoinst/openQA/pull/3534

Please keep in mind, even if this is not explictly mentioned: We probably all agree that without automatic tests we would not call any contributions properly long-term supportable. These tests could be very simple, e.g. something like podman run --rm -it .... openqa-worker --help or something. But if you do not plan to add tests as part of this ticket which is of course ok then please create a follow-up so that we don't forget that.

Last changes on the PR works on the testing package in my home https://build.opensuse.org/package/show/home:ilausuch:branches:devel:openQA/openQA_container_image_worker_x86

okurz wrote:

Please keep in mind, even if this is not explictly mentioned: We probably all agree that without automatic tests we would not call any contributions properly long-term supportable. These tests could be very simple, e.g. something like podman run --rm -it .... openqa-worker --help or something. But if you do not plan to add tests as part of this ticket which is of course ok then please create a follow-up so that we don't forget that.

Ack. The specific AC however had build and publish in OBS in it (which #43718 basically is), and we don't even know that we can run tests in OBS after building the image. Although adding two lines in our existing setup might be more straightforward.

unfortunately it seems you overlooked #43715#note-37 . I created #43706 for that now.