QA » openQA Project

I will start preparing a config for OSD now, the same way as i did for O3. Nginx will be deployed in parallel to Apache2, running on ports 8080 and 4433. Once i'm confident that the config works, i will wait for a slow day, and then do a switch where i stop Apache and run Nginx in its place. And finally once all issues have been resolved, Apache will be replaced with Nginx in Salt (i will probably need help there though). I don't expect this to go fast, since i want to be very careful about avoiding downtime.

Opened a PR with improvements for the default Nginx config from the O3 deployment: https://github.com/os-autoinst/openQA/pull/5185

kraih wrote:

I will start preparing a config for OSD now, the same way as i did for O3. Nginx will be deployed in parallel to Apache2, running on ports 8080 and 4433. Once i'm confident that the config works, i will wait for a slow day, and then do a switch where i stop Apache and run Nginx in its place. And finally once all issues have been resolved, Apache will be replaced with Nginx in Salt (i will probably need help there though). I don't expect this to go fast, since i want to be very careful about avoiding downtime.

For OSD I suggest to prepare changes in https://gitlab.suse.de/openqa/salt-states-openqa and not apply any manual changes. Regarding applying the actual changes I suggest to just look into https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls and search for "apache" and apply the corresponding necessary changes from nginx. Having the nginx config files coming from the openQA upstream repo should make that straightforward.

Can I blame the switch to nginx for https://github.com/os-autoinst/os-autoinst-distri-openQA/actions/runs/5178791698/jobs/9330745528?pr=121#step:5:240 reporting "502 Bad Gateway"? I doubt it's just a coincidence :)

okurz wrote:

EDIT: Interestingly https://openqa.opensuse.org/tests?resultfilter=Incomplete only showed the one job https://openqa.opensuse.org/tests/3338256 failing for the same reason, currently no other.

Seems like that was a one-off. The same job is now looking fine and I also couldn't find other instances of the API failure.

okurz wrote:

Can I blame the switch to nginx for https://github.com/os-autoinst/os-autoinst-distri-openQA/actions/runs/5178791698/jobs/9330745528?pr=121#step:5:240 reporting "502 Bad Gateway"? I doubt it's just a coincidence :)

Yes, the problem was that nginx can handle a whole lot more connections than Apache or our app server. It happily bombarded the app server with way too many requests during peak times. The new adjusted config with upstream max_conns limit appears to have taken care of the problem:

# The "max_conns" value should be identical to the maximum number of
# connections the webui is configured to handle concurrently
upstream webui {
    zone upstream_webui 64k;
    server 127.0.0.1:9526 max_conns=30;
}

upstream websocket {
    server 127.0.0.1:9527;
}

upstream livehandler {
    server 127.0.0.1:9528;
}

server {
    listen      80;
    server_name openqa.opensuse.org openqa.infra.opensuse.org;

    root /usr/share/openqa/public;

    client_max_body_size 0;
    client_body_buffer_size 64k;

    location /assets {
        alias /var/lib/openqa/share/factory;
        tcp_nopush         on;
        sendfile           on;
        sendfile_max_chunk 1m;
    }

    location /image {
        alias /var/lib/openqa/images;
        tcp_nopush         on;
        sendfile           on;
        sendfile_max_chunk 1m;
    }

    location /api/v1/ws/ {
        proxy_pass http://websocket;
        proxy_http_version 1.1;
        proxy_read_timeout 3600;
        proxy_send_timeout 3600;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location /liveviewhandler/ {
        proxy_pass http://livehandler;
        proxy_http_version 1.1;
        proxy_read_timeout 3600;
        proxy_send_timeout 3600;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location / {
        proxy_pass "http://webui";
        tcp_nodelay        on;
        proxy_read_timeout 900;
        proxy_send_timeout 900;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto "https";
    }

    access_log /space/logs/nginx/openqa.access_log;
    error_log /space/logs/nginx/openqa.error_log;
}

okurz wrote:

For OSD I suggest to prepare changes in https://gitlab.suse.de/openqa/salt-states-openqa and not apply any manual changes. Regarding applying the actual changes I suggest to just look into https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls and search for "apache" and apply the corresponding necessary changes from nginx. Having the nginx config files coming from the openQA upstream repo should make that straightforward.

Not sure i can work it out that way. I can try to make the openQA upstream Nginx config more reusable, but i'll probably not be able to figure out SSL without testing on the actual machine.

Opened a PR to update the default nginx config again: https://github.com/os-autoinst/openQA/pull/5191

I've checked the O3 logs again today for unexpected 502 errors. There were two larger batches, one at 07:44 and one at 11:44. But each corresponded with a restart of openqa-webui, so nothing to worry about. Looks like everything is still working as expected.

Today i've lowered the error_log level to warn, which showed another setting that can be optimised:

2023/06/08 12:12:18 [warn] 19902#19902: *8368071 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003363, client: 192.168.112.17, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345916/status HTTP/1.1", host: "openqa1-opensuse"
2023/06/08 12:12:19 [warn] 19902#19902: *8368086 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003364, client: 192.168.112.11, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345811/artefact HTTP/1.1", host: "openqa1-opensuse"
2023/06/08 12:12:19 [warn] 19902#19902: *8368099 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003365, client: 192.168.112.11, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345811/artefact HTTP/1.1", host: "openqa1-opensuse"
2023/06/08 12:12:19 [warn] 19902#19902: *8368121 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003366, client: 192.168.112.11, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345811/artefact HTTP/1.1", host: "openqa1-opensuse"
2023/06/08 12:12:19 [warn] 20748#20748: *8368149 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003367, client: 192.168.112.11, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345811/artefact HTTP/1.1", host: "openqa1-opensuse"
2023/06/08 12:12:20 [warn] 20748#20748: *8368205 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003368, client: 192.168.112.11, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345811/artefact HTTP/1.1", host: "openqa1-opensuse"
2023/06/08 12:12:20 [warn] 20748#20748: *8368226 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003369, client: 192.168.112.11, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345811/artefact HTTP/1.1", host: "openqa1-opensuse"
2023/06/08 12:13:15 [warn] 20748#20748: *8375999 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003370, client: 192.168.112.17, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345855/artefact HTTP/1.1", host: "openqa1-opensuse"
2023/06/08 12:13:33 [warn] 20746#20746: *8378374 a client request body is buffered to a temporary file /var/lib/nginx/tmp//0000003371, client: 192.168.112.18, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345643/artefact HTTP/1.1", host: "openqa1-opensuse"

This happens with a very high frequency, because we've only set client_body_buffer_size 64k;, which is too low to cover the 1mb chunks of artefact uploads we are usually dealing with. As a solution i've now configured client_body_buffer_size 2mb;. It won't completely eliminate buffering to files, since we still have larger uploads, but it should be enough to eliminate buffering for the vast majority of uploads.

Also experimented with proxy_request_buffering off, which completely disables upload buffering to files in Nginx. But that results in the app server getting overwhelmed in some cases:

2023/06/08 13:17:19 [error] 4625#4625: *309604 no live upstreams while connecting to upstream, client: 192.168.112.17, server: openqa.opensuse.org, request: "POST /api/v1/jobs/3345784/status HTTP/1.1", upstream: "http://webui/api/v1/jobs/3345784/status", host: "openqa1-opensuse"

192.168.112.17 - - [08/Jun/2023:13:17:19 +0000] "POST /api/v1/jobs/3345784/status HTTP/1.1" 502 157 "-" "Mojolicious (Perl)" rt=0.000 urt="0.000"

I find this quite interesting, because with buffering enabled we've not seen this before. That suggests the buffering does a pretty good job protecting the app server and we will be better off with client_body_buffer_size 2mb; in production.

Opened a PR for the buffer settings: https://github.com/os-autoinst/openQA/pull/5196