QA » openQA Project

Jan found an Nginx feature that would probably work for limiting concurrent uploads with the right location settings for the API endpoint. Log file upload retry should just work if rate limited, asset uploads are not so clear, there is a small possibility that they could get lost. So this needs to be tested.

I'll do some local experiments to figure out the right nginx settings. For production deployment we'll have to consider other factors too though, such as logging with rotation and TLS certificates. So there will probably be a follow up ticket for proper nginx deployment, if this approach works out for the rate limiting.

The nginx connection limits to appear to work exactly as we expected and are pretty straight forward to configure:

# Here we create two connection counters, one for all incoming
# connections, and one per IP address
limit_conn_zone $server_name zone=all:10m;
limit_conn_zone $binary_remote_addr zone=addr:10m;

server {
    listen       8080;
    server_name  localhost;
    root /usr/share/openqa/public;
    client_max_body_size 0;

    location /api/v1/ws/ {
        proxy_pass http://[::1]:9527;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location /liveviewhandler/ {
        proxy_pass http://[::1]:9528;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location /tests {
        # Here we limit the actual number of connections
        limit_conn all 4;
        limit_conn addr 2;

        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass "http://[::1]:9526/tests";
    }

    location / {
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass "http://[::1]:9526";
    }
}

For testing purposes i've tried to only put limits on /tests, 4 connections overall from all IP addresses, and 2 connections per individual IP address. And this config does exactly that.

And this one should do the trick for specifically rate limiting artefact uploads:

limit_conn_zone $server_name zone=servers:10m;

server {
    listen       8080;
    server_name  localhost;
    root /usr/share/openqa/public;
    client_max_body_size 0;

    location /api/v1/ws/ {
        proxy_pass http://[::1]:9527;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location /liveviewhandler/ {
        proxy_pass http://[::1]:9528;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location ~ ^/jobs/[0-9]+/artefact$ {
        limit_conn servers 50;

        proxy_set_header X-Rate-Limit-Test "true";
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass "http://[::1]:9526";
    }

    location / {
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass "http://[::1]:9526";
    }
}

With the rate limiting out of the way, now it is time to figure out all the other settings we would need for live deployment on O3/OSD.

For starters i'm setting up an Nginx setup on O3 in parallel to the current Apache2. Then we can do a switch over on a not so busy day to see if everything works. There's a whole lot of small workarounds in the Apache config, so probably there will be some minor breakage, but shouldn't be anything that we can't fix quickly too.

Nginx is now running on O3 on port 8080 and should work as an Apache replacement.Only remaining question is how we go about testing it in production without causing too serious disruptions.

I've set it up so starting and stopping the apache2 and nginx services should be enough to switch between the two reverse proxies.

$ cat /etc/nginx/conf.d/openqa.conf
server {
    listen      80;
    server_name openqa.opensuse.org openqa.infra.opensuse.org;

    root /usr/share/openqa/public;

    client_max_body_size 0;
    client_body_buffer_size 64k;
    client_header_buffer_size 4k;

    location /nginx_status {
        stub_status;
        access_log off;
        allow 127.0.0.1;
        deny all;
    }

    location /assets {
        alias /var/lib/openqa/share/factory;
        tcp_nopush         on;
        sendfile           on;
        sendfile_max_chunk 1m;
    }

    location /image {
        alias /var/lib/openqa/images;
        tcp_nopush         on;
        sendfile           on;
        sendfile_max_chunk 1m;
    }

    location /api/v1/ws/ {
        proxy_pass http://127.0.0.1:9527;
        proxy_http_version 1.1;
        proxy_read_timeout 3600;
        proxy_send_timeout 3600;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location /liveviewhandler/ {
        proxy_pass http://127.0.0.1:9528;
        proxy_http_version 1.1;
        proxy_read_timeout 3600;
        proxy_send_timeout 3600;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location / {
        proxy_pass "http://127.0.0.1:9526";
        tcp_nodelay        on;
        proxy_read_timeout 900;
        proxy_send_timeout 900;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto "https";
    }

    access_log /space/logs/nginx/openqa.access_log;
    error_log /space/logs/nginx/openqa.error_log;
}

Edit: Some more tuning was required on O3. Enabling sendfile for asset downloads turned out to be extremely important. Without it the whole webui felt sluggish, because downloads were slowing everything else down.

I've also set up logrotate based on the apache config.

/space/logs/nginx/openqa.access_log /space/logs/nginx/access_log {
    compress
    dateext
    delaycompress
    maxage 365
    rotate 10
    size=+4096k
    notifempty
    missingok
    create 644 root root
    sharedscripts
    postrotate
     systemctl reload nginx.service
     sleep 60
    endscript
}

/space/logs/nginx/openqa.error_log /space/logs/nginx/error_log {
    compress
    dateext
    delaycompress
    maxage 365
    rotate 10
    size=+4096k
    notifempty
    missingok
    create 644 root root
    sharedscripts
    postrotate
     systemctl reload nginx.service
     sleep 60
    endscript
}

Sounds great! Though I suggest to focus on nginx related work in #129490 and focus this ticket on a limit in pure openQA. In the end we might not want to continue this feature work in openQA if nginx helps us already to achieve the same as you have already found out

okurz wrote:

Sounds great! Though I suggest to focus on nginx related work in #129490 and focus this ticket on a limit in pure openQA. In the end we might not want to continue this feature work in openQA if nginx helps us already to achieve the same as you have already found out

That is exactly what we discussed and what Sebastian is doing here. It makes little sense to do the other way around.

cdywan wrote:

That is exactly what we discussed and what Sebastian is doing here. It makes little sense to do the other way around.

I don't see that it makes little sense to do what this ticket suggests. But in any case please make the work explicit and transparent by using #129490 for nginx related work

okurz wrote:

cdywan wrote:

That is exactly what we discussed and what Sebastian is doing here. It makes little sense to do the other way around.

I don't see that it makes little sense to do what this ticket suggests. But in any case please make the work explicit and transparent by using #129490 for nginx related work

I was talking about looking into nginx first ;-) Not sure how the tickets got mixed up again. We also had some confusion due to #129487. It seems we need to take better care of tickets with overlapping goals.

okurz wrote:

In the end we might not want to continue this feature work in openQA if nginx helps us already to achieve the same as you have already found out

To sum up the situation so far: Nginx performs very well in production on O3 (with the right tuning), and it does have all the rate/connection limiting features we need for fine grained control over worker uploads. So it would make sense to rephrase the ACs of this ticket and to block it on #129490 as a prerequisite for connection limiting.

kraih wrote:

So it would make sense to rephrase the ACs of this ticket and to block it on #129490 as a prerequisite for connection limiting.

I would rather put it like this: If we manage to use nginx to do all what we want to ensure that no openQA instance is overloaded then we don't need this openQA feature at all. It might still be slightly beneficial in non-nginx scenarios if there are any useful ones.

first #131024

https://github.com/os-autoinst/openQA/pull/5276 Limit number of running jobs per webui instance

https://github.com/os-autoinst/openQA/pull/5276 merged.
Another suggestion was to add a notice to the "All tests" page if the number of running jobs is currently at the limit, so instead of the normal "56 jobs are running" users would see something like "250 jobs are running (limit of running jobs reached)"

tinita wrote:

Another suggestion was to add a notice to the "All tests" page if the number of running jobs is currently at the limit, so instead of the normal "56 jobs are running" users would see something like "250 jobs are running (limit of running jobs reached)"

How about
"250 jobs are running (limited by server config)"

https://github.com/os-autoinst/openQA/pull/5279 Show max running jobs on /tests page

So I investigated the problem with max_running_jobs not always being set to the default value, causing warnings.
That is actually a problem of the tests, and it wouldn't happen in the productive code.
The problem is that apparently OpenQA::Setup::read_config is not always called when we create app instances in tests.

I think that's unfortunate because we have to cover for that in code by ensuring a default value in several places, although that should be done by OpenQA::Setup in just one place.
It would be good to make sure in tests we read the config and fill in defaults.

I will create a ticket because I had a quick look, and it looks to me like it's not fixable very fast: #134114

https://github.com/os-autoinst/openQA/pull/5285 merged and deployed on osd, so limit should now be effective.

https://github.com/os-autoinst/openQA/pull/5287 Remove defaults, should be ensured by OpenQA::Setup already - merged, deployed on o3

Now monitoring what's happening

Draft https://github.com/os-autoinst/openQA/pull/5289

I was wondering why the schedule method takes two additional parameters, as I can't find any code under lib that passes these:

sub schedule ($self, $allocated_workers = {}, $allocated_jobs = {}) {

I found https://github.com/os-autoinst/openQA/pull/3741/files and it looks like the parameters were added to be able to check the contents of those hashes (or actually only $allocated_workers) in a unit test.

Since I'm extracting _allocate_jobs this might not be necessary anymore, and I can remove those parameters.

Still working on the test.
One problem is that 04-scheduler.t so far only had one worker, so there are no tests yet that deal with more than one job assignment.

Ready for review: https://github.com/os-autoinst/openQA/pull/5289 Make max_running_jobs a hard limit

PR approved. As mentioned in https://github.com/os-autoinst/openQA/pull/5289#issuecomment-1713488199 please look into the runtime improvement of the unit test and further refactoring of the code as discussed.

okurz wrote in #note-51:

PR approved. As mentioned in https://github.com/os-autoinst/openQA/pull/5289#issuecomment-1713488199 please look into the runtime improvement of the unit test and further refactoring of the code as discussed.

I would lean towards a follow-up ticket here as this has already been in the queue for a bit and probably won't be done today... of course we can discuss this in the Unblock.

https://github.com/os-autoinst/openQA/pull/5306 scheduler: Log statistics of rejected jobs (#135578)

https://github.com/os-autoinst/openQA/pull/5306 merged

https://github.com/os-autoinst/openQA/pull/5315 Reduce runtime of t/04-scheduler.t

https://github.com/os-autoinst/openQA/pull/5315 merged, what's next/missing?