QA » openQA Project

I think logically this will come after #89719 hence marking this as Blocked, since the AC assume an existing compose test that can be extended

cdywan wrote:

I think logically this will come after #89719 hence marking this as Blocked, since the AC assume an existing compose test that can be extended

Correction. Since @okurz filed a separate issue this can actually be blocked by #89722

I checked that the list of running containers could be checked with
docker-compose ps --service --filter status=running

Unfortunately I checked using status=exited but only works on docker but not on docker-compose (At least in the version 1.27.4)

An other option is to use docker-compose ps | grep Exit but this has the problem that none of the container have a name containing Exit

ilausuch wrote:

I checked that the list of running containers could be checked with
docker-compose ps --service --filter status=running

Unfortunately I checked using status=exited but only works on docker but not on docker-compose (At least in the version 1.27.4)

An other option is to use docker-compose ps | grep Exit but this has the problem that none of the container have a name containing Exit

I would approach it from the mojo app first before looking at the containers. If e.g. database access is not locked that would be a problem for other use cases, too.

Target version set to future

From the description it seemed like a pretty serious bug to me. Did you confirm that it's not?

No, I did not. We did the exception to originally accept the docker-compose.yaml without automatic tests. So we just learned again what we did know anyway: Every code contribution needs an automatic test. Gladly the docker-compose approach is not prominently mentioned in documentation yet.

So #89722 first

okurz wrote:

No, I did not. We did the exception to originally accept the docker-compose.yaml without automatic tests. So we just learned again what we did know anyway: Every code contribution needs an automatic test. Gladly the docker-compose approach is not prominently mentioned in documentation yet.

So #89722 first

Sure. I just thought that's conveyed by the blocked status and not necessary to handle manually.

I ran a test that fails
https://github.com/os-autoinst/openQA/pull/3818/checks?check_run_id=2236273686
I only did a sleep before check that all the containers are UP to grant time to the failure.

There is a draft PR with some approaches
https://github.com/os-autoinst/openQA/pull/3821

The problem that I have detected is that all the webui containers try to create and initialize the DB at the same time, and since all the webui replicas are launched at the same time by the docker-compose and we have more than one, it generates the problem
My approach has been to create an initial webui and sort the dependencies so that the webui cluster is only initialized after initializing this first webui.
In addition, I base the dependencies on the health checks ensuring that it does not go to the next phase unless the previous one is complete.

Why it takes >30min?

I think the process spend a lot of time building the images. And in empirical test in my local machine, the installation of packages is the most tedious part. But I think this should measured to be sure.

One idea:
We need to build these images because maybe the PR that is launching the test do some changes that should be tested. How ever, we could base these images in a previous build image where all the packages are already installed. This will accelerate the process of testing. At least it is something that I have already tried in my experience

ilausuch wrote:

Why it takes >30min?

Was that comment meant for #90767 ?

We have here a new situation under investigation
https://github.com/os-autoinst/openQA/pull/3850/checks?check_run_id=2388942783

Webui_1 is unhealthy

        Name                       Command                   State                                        Ports                                 
------------------------------------------------------------------------------------------------------------------------------------------------
webui_db_1              docker-entrypoint.sh postgres    Up (healthy)     5432/tcp                                                              
webui_webui_1           /root/run_openqa.sh              Up (unhealthy)   443/tcp, 80/tcp, 0.0.0.0:49155->9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp
webui_webui_2           /root/run_openqa.sh              Up (healthy)     443/tcp, 80/tcp, 0.0.0.0:49154->9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp
webui_webui_db_init_1   sh -c chmod -R a+rwX /data ...   Up (healthy)     443/tcp, 80/tcp, 0.0.0.0:49153->9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp

Solved with https://github.com/os-autoinst/openQA/pull/3850

ilausuch wrote:

Solved with https://github.com/os-autoinst/openQA/pull/3850

So is #91377 a duplicate of this? Or vice versa? Is the same problem being solved here?

No, #91377 was for the static check and this if for the docker-compose test

The AC1 is solved with https://github.com/os-autoinst/openQA/pull/3821

About the AC2 all the container should be with a healthy status.

But we can see here that nginx (that is the last container in the chain) needs some time to stand-up.

https://github.com/os-autoinst/openQA/pull/3864/checks?check_run_id=2444880438

        Name                       Command                       State                                                     Ports                                          
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
webui_db_1              docker-entrypoint.sh postgres    Up (healthy)            5432/tcp                                                                                 
webui_gru_1             sh -c /root/run_openqa.sh| ...   Up (healthy)            443/tcp, 80/tcp, 9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp                                  
webui_livehandler_1     /root/run_openqa.sh              Up (healthy)            443/tcp, 80/tcp, 9526/tcp, 9527/tcp, 0.0.0.0:9528->9528/tcp,:::9528->9528/tcp, 9529/tcp  
webui_nginx_1           /entrypoint.sh                   Up (health: starting)   0.0.0.0:9526->9526/tcp,:::9526->9526/tcp                                                 
webui_scheduler_1       /root/run_openqa.sh              Up (healthy)            443/tcp, 80/tcp, 9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp                                  
webui_websockets_1      /root/run_openqa.sh              Up (healthy)            443/tcp, 80/tcp, 9526/tcp, 0.0.0.0:9527->9527/tcp,:::9527->9527/tcp, 9528/tcp, 9529/tcp  
webui_webui_1           /root/run_openqa.sh              Up (healthy)            443/tcp, 80/tcp, 0.0.0.0:49155->9526/tcp,:::49155->9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp
webui_webui_2           /root/run_openqa.sh              Up (healthy)            443/tcp, 80/tcp, 0.0.0.0:49154->9526/tcp,:::49154->9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp
webui_webui_db_init_1   sh -c chmod -R a+rwX /data ...   Up (healthy)            443/tcp, 80/tcp, 0.0.0.0:49153->9526/tcp,:::49153->9526/tcp, 9527/tcp, 9528/tcp, 9529/tcp

I am thinking in two possible solutions

• Give some time to nginx to start with an active waiting until healthy or unhealthy that are "final statuses"
• Create a dummy container dependent of nginx to ensure that nginx has started with healthy

The AC2 has moved to #91815. As far as AC1 is covered (https://progress.opensuse.org/issues/89731#note-27) this ticket can be considered resolved. do you Agree?