QA (public) » openQA Project (public)

start with the webui server

The web UI server runs a lot of services and provides the filesystem /var/lib/openqa so this raises a lot of questions:

1. I assume there would still be just one database.
2. What about files within /var/lib/openqa? I suppose this file system still needs to be shared between the web UIs, e.g. using NFS like we already share /var/lib/openqa/share with the worker hosts.
3. What about the scheduler? I suppose there should still be just one to avoid race conditions.
4. What about the liveviewhandler and websocket server? Both are so far separate services because only a single process is supposed to serve the routes they provide because they rely on resources which needed to be shared between multiple instances if there were multiple instances. In case of the liveviewhandler that resource is websocket connection to another server so sharing it would be quite hard. So there should be only one of these services (in their current form).
5. What about the Minion service? Since we rely on locking to avoid running jobs in parallel which shouldn't run in parallel I'd say there should only be one Minion service as well or multiple Minion services would share the same database. These jobs often involve a lot filesystem access under /var/lib/openqa so likely having multiple Minion services (sharing the same database) is not very beneficial if these filesystem access would then be performed from a remote host.

Looks like we would really only run multiple instances of the openqa-webui.service and keep the other services on the web UI host as they are. Currently we're actually running multiple web UI processes via preforking. Maybe we could reduce the number of prefork processes then. Otherwise we would likely need to take care not to exceed the limit for PostgreSQL connections, e.g. by implementing #55262.

can cause longer downtimes and make upgrades of OS more risky, e.g. when we do not have management access to VMs that might fail to reboot.

But some host needs to provide the "entry point" and database, right? So if that host is down we would still have a downtime.

Also, load-balancing can help as well as having switch-over deployments possible for easier testing, staging, etc.

As long as we share the same database I would be very careful to run multiple instances of different openQA versions in parallel.

mkittler wrote:

start with the webui server

The web UI server runs a lot of services and provides the filesystem /var/lib/openqa so this raises a lot of questions:

1. I assume there would still be just one database.

either that or database replication

1. What about files within /var/lib/openqa? I suppose this file system still needs to be shared between the web UIs, e.g. using NFS like we already share /var/lib/openqa/share with the worker hosts.

Yes, I do not see an immediate problem with that as we already have either scalability features in place, like minion jobs with locks, or the same problems we want to fix anyway, like when saving needles interferes with updating the git repo with fetchneedles.

1. What about the scheduler? I suppose there should still be just one to avoid race conditions.

yes

1. What about the liveviewhandler and websocket server? Both are so far separate services because only a single process is supposed to serve the routes they provide because they rely on resources which needed to be shared between multiple instances if there were multiple instances. In case of the liveviewhandler that resource is websocket connection to another server so sharing it would be quite hard. So there should be only one of these services (in their current form).

yes. This is why it is good that we have already separate services even though we do not make full use of the possibilities yet (like running multiple instances of webui service but not liveviewhandler and websocket)

1. What about the Minion service? Since we rely on locking to avoid running jobs in parallel which shouldn't run in parallel I'd say there should only be one Minion service as well or multiple Minion services would share the same database. These jobs often involve a lot filesystem access under /var/lib/openqa so likely having multiple Minion services (sharing the same database) is not very beneficial if these filesystem access would then be performed from a remote host.

Looks like we would really only run multiple instances of the openqa-webui.service and keep the other services on the web UI host as they are. Currently we're actually running multiple web UI processes via preforking. Maybe we could reduce the number of prefork processes then. Otherwise we would likely need to take care not to exceed the limit for PostgreSQL connections, e.g. by implementing #55262.

exactly, same as now run multiple processes but potentially spread out over different hosts and be able to dynamically alter the number and control them individually, e.g. for blue/green deployments.

can cause longer downtimes and make upgrades of OS more risky, e.g. when we do not have management access to VMs that might fail to reboot.

But some host needs to provide the "entry point" and database, right? So if that host is down we would still have a downtime.

yes, but this host can be the one used for the http load-balancer which are more simple than openQA itself, have already built-in scalability features and can monitored for example by kubernetes itself.

[…]
As long as we share the same database I would be very careful to run multiple instances of different openQA versions in parallel.

yes, careful but I would still do it :) If you like we could have a simple experimentation task first which would be "run two openQA webui instances of different versions against a single database". But I guess with just some simple practices how we introduce schema changes we should be good. If not, then just shut off all old versions as soon as the schema is newer than the currently supported one.

If you like we could have a simple experimentation task first which would be "run two openQA webui instances of different versions against a single database".

We don't need an experimentation task. I can already tell you that it will sometimes work and sometimes now. If both versions use the same schema version it will work and otherwise not. The version with the newer schema will obviously migrate the database to that new version. The old version will run into database errors then. I know that too well because I often switch between different versions locally which use a different schema version.

Initial approach
https://github.com/os-autoinst/openQA/pull/3431

Right now I am covering all the cases that has being mentioned on the PR review and also

• removing the dependence on apache
• adding a loadbalancer for the webui
• configuring webui, livehandler, websockets, gru and scheduler as independent containers

webui_data_1          /bin/sh -c /usr/bin/tail - ...   Up
webui_db-admin_1      entrypoint.sh docker-php-e ...   Up      0.0.0.0:8080->8080/tcp
webui_db_1            docker-entrypoint.sh postgres    Up      5432/tcp
webui_gru_1           /root/run_openqa.sh              Up      9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp
webui_haproxy_1       /docker-entrypoint.sh hapr ...   Up      0.0.0.0:9526->9526/tcp
webui_livehandler_1   /root/run_openqa.sh              Up      9526/tcp, 9527/tcp, 0.0.0.0:9528->9528/tcp, 9529/tcp
webui_scheduler_1     /root/run_openqa.sh              Up      9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp
webui_websockets_1    /root/run_openqa.sh              Up      9526/tcp, 0.0.0.0:9527->9527/tcp, 9528/tcp, 9529/tcp
webui_webui_1         /root/run_openqa.sh              Up      0.0.0.0:32813->9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp
webui_webui_2         /root/run_openqa.sh              Up      0.0.0.0:32814->9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp

Some changes done in the PR. If you like it I'll proceed to document this version

Added the documentation to the README.md

A question, should it be retro-compatible to run without docker-compose?
In case yes, I create a new commit that allows this retro-compatibility, so if you don't use the docker-compose is expected the same behavior as before.

Before to close this task I would like to check the compatibility with the workers working in this ticket #43715

This ticket is bloqued until the mistery of https://progress.opensuse.org/issues/43715#note-22 will be solved

Problems solved with the worker. Now it is using nginx as LB. Preparing the auto scaling and the PR

https://github.com/os-autoinst/openQA/pull/3488

This PR has the same purpose than previous one but using nginx to be able to integrate the websocket, livehandler in the path structure of the API.
This solves the communication from the workers with the api and the websockets using the same port

Thank you. To provide a little bit more information: We have a proof-of-concept with the instructions so far in https://github.com/os-autoinst/openQA/blob/master/docker/README.md . What we can do as next steps is to make sure this is included in https://open.qa/docs . Not in this ticket though, included now in #76990