QA » openQA Project

This is indeed very much related to #129619 and can probably be solved with a few nginx connection limit settings too. The only real concern is that the asset upload retry code might not be good enough yet.

This nginx config should work for rate limiting artefact uploads:

limit_conn_zone $server_name zone=servers:10m;

upstream webui {
    zone upstream_webui 64k;
    server [::1]:9526 max_conns=30;
}

server {
    listen       8080;
    server_name  localhost;
    root /usr/share/openqa/public;
    client_max_body_size 0;

    location /api/v1/ws/ {
        proxy_pass http://[::1]:9527;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location /liveviewhandler/ {
        proxy_pass http://[::1]:9528;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    # Limit artefact uploads to 50 connections
    location ~ ^/jobs/[0-9]+/artefact$ {
        proxy_pass "http://webui";
        limit_conn servers 50;

        proxy_set_header X-Rate-Limit-Test "true";
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location / {
        proxy_pass "http://webui";
        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

okurz wrote:

I agree that we should first await full results from #129490 before we look into this ticket. I assume you wanted to block on that?

Yea, and then i had an idea for a generic connection limit feature that could work on the Mojolicious routing level.

$job_r->limit_connections(50)->post('/artefact')->name('apiv1_create_artefact')->to('job#create_artefact');

It's actually pretty simple to implement and could be applied to any route with ->limit_connections(50). It would use a Minion shared lock in PostrgeSQL and respond with a 503 if the limit is exceeded. Basically emulating how nginx rate limits work in a less efficient way. So it would be useful to test the basic concept of limiting artefact uploads based on number of connections, but we'd throw the code away again in the future.

Nice idea and sounds promising but as you said we would maybe throw away the code again so that's why I suggest to wait for the nginx story.

okurz wrote:

Nice idea and sounds promising but as you said we would maybe throw away the code again so that's why I suggest to wait for the nginx story.

It might be worth trying because i still suspect that the whole concept will not work, because rejecting already uploaded chunks of data will cause extra bandwidth to be used. Or bad timing with rejections could cause times where available bandwidth is used less efficiently than it is now.

okurz wrote:

I agree that we should first await full results from #129490 before we look into this ticket. I assume you wanted to block on that?

Clarifying for myself, but also commenting here to make it clear for others. The blocker has been resolved. There's #130636 which isn't on the backlog but we're effectively waiting on #131024 now.