openSUSE admin

Looking at /var/log/nginx/error.logs, I see e.g.

/var/log/nginx/error.log-20221014.xz:2022/10/13 13:43:35 [error] 2062#2062: *624197 client intended to send too large body: 5071396 bytes, client: 127.0.0.1, server: lists.opensuse.org, request: "POST /archives/api/mailman/archive HTTP/1.1", host: "localhost"

This started on 14 October and has been going on ever since. Was some limit reset? As far as I can tell, we have "client_max_body_size 400M;", but that obviously does not work, somehow.

I also messages like this (when trying to discard a message):

2022/12/14 13:05:36 [error] 8935#8935: *5233 upstream prematurely closed connection while reading response header from upstream, client: 2a03:7520:4c68:1:ff99:ffff:0:98fc, server: lists.opensuse.org, request: "POST /manage/lists/sourcedvd.lists.opensuse.org/held_messages HTTP/1.1", upstream: "http://127.0.0.1:8000/manage/lists/sourcedvd.lists.opensuse.org/held_messages", host: "lists.opensuse.org", referrer: "https://lists.opensuse.org/manage/lists/sourcedvd.lists.opensuse.org/held_messages"

Upstream being http://127.0.0.1:8000 - that is gunicorn.

In /var/log/postorius/gunicorn.log, I see numerous "[CRITICAL] WORKER TIMEOUT".

pjessen wrote:

This started on 14 October and has been going on ever since. Was some limit reset? As far as I can tell, we have "client_max_body_size 400M;", but that obviously does not work, somehow.

I have reduced to "client_max_body_size 10M;" and this seems to work ??

Have changed the gunicorn timeout to 0, over the commandline, with mailman-web.service.d/timeout.conf. Instead of a 502, I'm now getting a 504, which suggests it it nginx timing out.

By default nginx has a 60 second timeout - I have added this to the http{} section in /etc/nginx/nginx.conf:

proxy_send_timeout          600;
proxy_read_timeout          600;
send_timeout                600;

This does the trick - why discarding a message takes about 90secs, I have no idea.

pjessen wrote:

I'll leave this for now and review next week.

So far it is looking good. The "worker timeout" messages have gone from the gunicorn log, but there are still some "upstream prematurely closed connection" messages in the nginx errorlog, interestingly all on exports of archives in mailbox format.

luc14n0 wrote:

Use MySQL for redirection based on table look-ups, instead of Nginx.

I don't see nginx and the huge redirection maps as being the culprit here, but it ought to be easy to prove/disprove.
Just a little earlier, I wanted to remove some non-members addresses from users.lists:

• After hitting "delete", it took 2min16sec for the Confirm page to appear
• after hitting "Confirm", it took 2m12sec to return to the list page.

pjessen wrote:

luc14n0 wrote:

Use MySQL for redirection based on table look-ups, instead of Nginx.

I don't see nginx and the huge redirection maps as being the culprit here, but it ought to be easy to prove/disprove.

I removed the include for thee mails redirects, almost 3million lines, did an "nginx reload" and tried to remove a non-member again:

• After hitting "delete", it took 2min8sec for the Confirm page to appear.
• after hitting "Confirm", it took 2min7sec to return to the list page.

luc14n0 wrote:

Use MySQL for redirection based on table look-ups, instead of Nginx.

FWIW, I wrote a small proxy daemon for that, see #101842. It reduced the memory footprint, but did nothing for this issue. Discarding a held message currently takes 3 minutes and 15 seconds.