Project

General

Profile

tickets #102203

mailman3 - uwsgi memory leak?

Added by pjessen 9 months ago. Updated about 2 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Mailing lists
Target version:
-
Start date:
2021-11-10
Due date:
% Done:

80%

Estimated time:

Description

Despite having increased memory on mailman3 from 8Gb to 12Gb, 'uwsgi' just uses up more of it:

mailman3 (lists.o.o):~ # dmesg -T | grep Out
[Sat Nov  6 00:13:46 2021] Out of memory: Killed process 1413 (uwsgi) total-vm:6920348kB, anon-rss:6703136kB, file-rss:0kB, shmem-rss:128kB
[Sat Nov  6 04:43:29 2021] Out of memory: Killed process 6469 (uwsgi) total-vm:8274960kB, anon-rss:8054432kB, file-rss:0kB, shmem-rss:128kB
[Sat Nov  6 10:09:43 2021] Out of memory: Killed process 11454 (uwsgi) total-vm:8244728kB, anon-rss:8042024kB, file-rss:0kB, shmem-rss:128kB
[Sat Nov  6 19:13:26 2021] Out of memory: Killed process 17945 (uwsgi) total-vm:8281008kB, anon-rss:8056436kB, file-rss:0kB, shmem-rss:128kB
[Sun Nov  7 00:00:45 2021] Out of memory: Killed process 29157 (uwsgi) total-vm:7231816kB, anon-rss:7007352kB, file-rss:0kB, shmem-rss:128kB
[Sun Nov  7 04:15:18 2021] Out of memory: Killed process 2782 (uwsgi) total-vm:8303620kB, anon-rss:8061684kB, file-rss:0kB, shmem-rss:128kB
[Sun Nov  7 08:47:29 2021] Out of memory: Killed process 7453 (uwsgi) total-vm:8182164kB, anon-rss:7980296kB, file-rss:0kB, shmem-rss:128kB
[Sun Nov  7 11:28:15 2021] Out of memory: Killed process 12774 (uwsgi) total-vm:8297484kB, anon-rss:8040232kB, file-rss:0kB, shmem-rss:128kB
[Sun Nov  7 15:05:22 2021] Out of memory: Killed process 16003 (uwsgi) total-vm:8259072kB, anon-rss:8029160kB, file-rss:0kB, shmem-rss:128kB
[Mon Nov  8 00:08:59 2021] Out of memory: Killed process 20442 (uwsgi) total-vm:6887800kB, anon-rss:6657480kB, file-rss:0kB, shmem-rss:128kB
[Mon Nov  8 03:43:51 2021] Out of memory: Killed process 31699 (uwsgi) total-vm:8273612kB, anon-rss:8023212kB, file-rss:0kB, shmem-rss:128kB
[Mon Nov  8 19:17:54 2021] Out of memory: Killed process 3302 (uwsgi) total-vm:8195640kB, anon-rss:7975640kB, file-rss:0kB, shmem-rss:128kB
[Tue Nov  9 00:14:57 2021] Out of memory: Killed process 22671 (uwsgi) total-vm:6814652kB, anon-rss:6553372kB, file-rss:0kB, shmem-rss:128kB
[Tue Nov  9 06:01:11 2021] Out of memory: Killed process 28922 (uwsgi) total-vm:7822680kB, anon-rss:7612088kB, file-rss:0kB, shmem-rss:128kB
[Wed Nov 10 00:01:29 2021] Out of memory: Killed process 3191 (uwsgi) total-vm:5780740kB, anon-rss:5549512kB, file-rss:0kB, shmem-rss:128kB

Related issues

Related to openSUSE admin - tickets #101842: mailman3 - nginx memory usage?New2021-11-02

Related to openSUSE admin - tickets #113111: lists.opensuse.org - webserver backand timeouts - nginx 504 gateway timeoutNew2022-06-28

History

#1 Updated by pjessen 9 months ago

  • Private changed from Yes to No

I suspect this is also making uwsgi slower and slower - today since midnight 1040 time-outs:

2021/11/10 10:09:38 [error] 25453#25453: *3843163 upstream timed out (110: Connection timed out) while connecting to upstream, client: 114.119.138.77, server: lists.opensuse.org, request: "GET /archives/list/users-hu@lists.opensuse.org/export/users-hu@lists.opensuse.org-2009-09.mbox.gz?start=2009-08-01&end=2009-09-01 HTTP/1.1", upstream: "uwsgi://0.0.0.0:8000", host: "lists.opensuse.org"

#2 Updated by pjessen 9 months ago

I did some googling and I have amended uwsgi.ini with this:

# Worker Management
max-requests = 1000                  ; Restart workers after this many requests
max-worker-lifetime = 3600           ; Restart workers after this many seconds
reload-on-rss = 1024                 ; Restart workers after this much resident memory
worker-reload-mercy = 60             ; How long to wait before forcefully killing workers

From https://www.techatbloomberg.com/blog/configuring-uwsgi-production-deployment/

#3 Updated by pjessen 9 months ago

I am not sure if the above has worked or not. There has not been any oom kill since I added it, but I have just now seen a uwsgi process with about 4.5Gb in use. It was replaced soon after though.

#4 Updated by pjessen 9 months ago

No, it works. I have just watched uwsgi being stopped and a new process started.

#5 Updated by pjessen 9 months ago

  • Category set to Mailing lists
  • Status changed from New to Workable
  • Assignee set to pjessen

I don't get it.
Today, four oom kills:

dmesg -T | grep Killed
[Thu Nov 18 15:22:36 2021] Out of memory: Killed process 19146 (uwsgi) total-vm:9138912kB, anon-rss:8921448kB, file-rss:0kB, shmem-rss:128kB
[Thu Nov 18 16:21:46 2021] Out of memory: Killed process 20832 (uwsgi) total-vm:9165396kB, anon-rss:8915852kB, file-rss:0kB, shmem-rss:128kB
[Thu Nov 18 16:49:57 2021] Out of memory: Killed process 21631 (uwsgi) total-vm:9147092kB, anon-rss:8916712kB, file-rss:0kB, shmem-rss:124kB
[Thu Nov 18 17:43:49 2021] Out of memory: Killed process 22753 (uwsgi) total-vm:9136628kB, anon-rss:8899684kB, file-rss:0kB, shmem-rss:124kB

#6 Updated by pjessen 9 months ago

18 November - 10 oom kills
19 November - 2 oom kills
20 November - 6 oom kills
21 November - 12 oom kills
22 November - 11 oom kills
23 November - 5 oom kills
24 November - 7 oom kills
25, 26 - none.
27 November - 1 oom kill
since then none.

#7 Updated by pjessen about 2 months ago

  • Status changed from Workable to In Progress
  • % Done changed from 0 to 80

In the last thirty days, only one oom kill. It seems clear that there is a memory leak, but at least this work-around (uwsgi.ini) seems to have taken care of the worst. Clearly this needs to be salted, but I'll admit I'm not very good with salt.

#8 Updated by pjessen about 2 months ago

#9 Updated by pjessen about 2 months ago

  • Related to tickets #113111: lists.opensuse.org - webserver backand timeouts - nginx 504 gateway timeout added

Also available in: Atom PDF