openSUSE admin

I have honestly no idea - I did a restart of mailman, let us see if that helps.

Well, it didn't help right away. I see the xapian filesystem is mounted, and it appears to be updated too. (recent timestamps)

That machine is not well - looking at dmesg output, the oom killer has been very active since at least 14 October, judging by the logs since end of September. At a quick glance, it is always uwsgi that is being killed.

The search index remains broken - searching for posts from today on factory.lists.o.o and the latest hits are six months old.
My restart clearly did not help. Like a week ago, the filesystem holding the xapian index is mounted, and timestamps are current except for position.glass, which is about 6 months old ... dated 21 April 2021.

I increased RAM of os-mailman3 from 8 to 12GB

Maybe mailman's uwsgi process has a memory leak. I saw it at 3GB used before the reboot.

pjessen wrote:

I won't pretend to understand much of this stuff, but in the logs I see this, beginning 19 October:

ERROR 2021-10-25 12:11:53,461 12586 django.request Internal Server Error: /archives/search
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/xapian_backend.py", line 27, in
import xapian
File "/usr/lib64/python3.6/site-packages/xapian/init.py", line 24, in
from . import _xapian
ImportError: /lib64/librt.so.1: symbol __pthread_attr_copy, version GLIBC_PRIVATE not defined in file libpthread.so.0 with link time reference

The last time this happened was 25 October, since then nothing. Heisenbug?

bmwiedemann wrote:

I increased RAM of os-mailman3 from 8 to 12GB

Maybe mailman's uwsgi process has a memory leak. I saw it at 3GB used before the reboot.

I was just now keeping an eye on it - it is at 2.8Gb, and taking up an entire CPU. Sometimes it drops down to 2.4G, but then quickly builds up again.

pjessen wrote:

bmwiedemann wrote:

I increased RAM of os-mailman3 from 8 to 12GB

Maybe mailman's uwsgi process has a memory leak. I saw it at 3GB used before the reboot.

I was just now keeping an eye on it - it is at 2.8Gb, and taking up an entire CPU. Sometimes it drops down to 2.4G, but then quickly builds up again.

Currently 'uwsgi' is at almost 6Gb:

top - 09:41:05 up 7 days, 18:20,  1 user,  load average: 0.56, 0.50, 0.45
Tasks: 194 total,   1 running, 193 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.5 us,  0.9 sy,  0.0 ni, 82.6 id,  0.2 wa,  0.0 hi,  0.6 si,  0.4 st
MiB Mem : 11754.97+total, 1635.348 free, 9605.910 used,  513.719 buff/cache
MiB Swap:  255.996 total,    1.520 free,  254.477 used. 1817.375 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                 
25647 mailman   20   0 6088584 5.611g  14436 S 38.67 48.87 219:24.02 uwsgi                                                                                   
25452 nginx     20   0 1414236 1.245g   2116 S 0.000 10.84  32:53.13 nginx                                                                                   
25456 nginx     20   0 1414236 1.244g   2080 S 0.000 10.83   0:08.64 nginx                                                                                   
25453 nginx     20   0 1414236 1.243g      8 S 0.333 10.82  13:42.09 nginx                                                                                   
25454 nginx     20   0 1414236 1.242g    200 S 0.000 10.82   0:36.08 nginx                                                                                   
25455 nginx     20   0 1414236 1.242g    840 S 0.000 10.82   0:15.16 nginx                                                                                   
25457 nginx     20   0 1414236 1.240g   2080 S 0.000 10.80   0:05.68 nginx                                                                                   
 1897 root      20   0 1401984 1.238g      4 S 0.000 10.78   1:49.58 nginx                                                                                  

For the apparent memory leak in uwsgi, see #102203

FWIW, the search index still appears to be being updated:

# l /var/lib/mailman_webui/xapian_index
total 133482760
drwxr-xr-x 2 mailman mailman         126 Dec  2 08:00 ./
drwxr-xr-x 6 mailman mailman        4096 Apr 21  2021 ../
-rw-r--r-- 1 mailman mailman  5250809856 Dec  2 08:00 docdata.glass
-rw-r--r-- 1 mailman mailman           0 Dec  2 08:00 flintlock
-rw-r--r-- 1 mailman mailman         183 Dec  2 08:00 iamglass
-rw-r--r-- 1 mailman mailman 85624954880 Dec  1 03:10 position.glass
-rw-r--r-- 1 mailman mailman 32488120320 Dec  2 08:00 postlist.glass
-rw-r--r-- 1 mailman mailman 13322428416 Dec  2 08:00 termlist.glass

However, searching on recent posts on e.g. factory does not show any recent results.

The search index still appears to be being updated:

l /var/lib/mailman_webui/xapian_index
total 133482760
drwxr-xr-x 2 mailman mailman         126 Jan 18 10:00 ./
drwxr-xr-x 6 mailman mailman        4096 Dec 16 14:35 ../
-rw-r--r-- 1 mailman mailman  5250809856 Jan 18 10:00 docdata.glass
-rw-r--r-- 1 mailman mailman           0 Jan 18 10:00 flintlock
-rw-r--r-- 1 mailman mailman         182 Jan 18 10:00 iamglass
-rw-r--r-- 1 mailman mailman 85624954880 Dec 14 14:46 position.glass
-rw-r--r-- 1 mailman mailman 32488120320 Jan 18 10:00 postlist.glass
-rw-r--r-- 1 mailman mailman 13322428416 Jan 18 10:00 termlist.glass

Hi, I just stumbled again over the broken search on the mailing list archives.
I understand Per (thanks for your effort!) has no further idea - how do we proceed? Asked in oS mailinglists?