tickets #100802
openList archive search broken?
Added by tejas.guruswamy@opensuse.org over 2 years ago. Updated over 1 year ago.
0%
Description
Hi heroes,
I cannot see any results from the lists.opensuse.org archive HyperKitty
search function newer than about ~4months. Is some database/indexing broken?
For example, there was a thread on opensuse-users about a diagram editor
last week, visible from the list front page:
But this search query does not find it at all:
Tejas
Updated by cboltz over 2 years ago
- Category set to Mailing lists
- Assignee set to pjessen
Updated by pjessen over 2 years ago
I have honestly no idea - I did a restart of mailman, let us see if that helps.
Well, it didn't help right away. I see the xapian filesystem is mounted, and it appears to be updated too. (recent timestamps)
Updated by pjessen over 2 years ago
That machine is not well - looking at dmesg output, the oom killer has been very active since at least 14 October, judging by the logs since end of September. At a quick glance, it is always uwsgi that is being killed.
Updated by pjessen over 2 years ago
The search index remains broken - searching for posts from today on factory.lists.o.o and the latest hits are six months old.
My restart clearly did not help. Like a week ago, the filesystem holding the xapian index is mounted, and timestamps are current except for position.glass, which is about 6 months old ... dated 21 April 2021.
Updated by pjessen over 2 years ago
- Private changed from Yes to No
I won't pretend to understand much of this stuff, but in the logs I see this, beginning 19 October:
ERROR 2021-10-25 12:11:53,461 12586 django.request Internal Server Error: /archives/search
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/xapian_backend.py", line 27, in <module>
import xapian
File "/usr/lib64/python3.6/site-packages/xapian/__init__.py", line 24, in <module>
from . import _xapian
ImportError: /lib64/librt.so.1: symbol __pthread_attr_copy, version GLIBC_PRIVATE not defined in file libpthread.so.0 with link time reference
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python3.6/site-packages/hyperkitty/views/search.py", line 54, in search
results = EmptySearchQuerySet()
File "/usr/lib/python3.6/site-packages/haystack/query.py", line 26, in __init__
self._determine_backend()
File "/usr/lib/python3.6/site-packages/haystack/query.py", line 59, in _determine_backend
self.query = connections[backend_alias].get_query()
File "/usr/lib/python3.6/site-packages/haystack/utils/loading.py", line 116, in __getitem__
self.connections_info[key]["ENGINE"]
File "/usr/lib/python3.6/site-packages/haystack/utils/loading.py", line 61, in load_backend
return import_class(full_backend_path)
File "/usr/lib/python3.6/site-packages/haystack/utils/loading.py", line 23, in import_class
module_itself = importlib.import_module(module_path)
File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/lib/python3.6/site-packages/xapian_backend.py", line 29, in <module>
raise MissingDependency("The 'xapian' backend requires the installation of 'Xapian'. "
haystack.exceptions.MissingDependency: The 'xapian' backend requires the installation of 'Xapian'. Please refer to the documentation.
It looks like glibc was updated on 19 October, but this ticket goes further back, so the above may not be related.
Updated by bmwiedemann over 2 years ago
I increased RAM of os-mailman3 from 8 to 12GB
Maybe mailman's uwsgi process has a memory leak. I saw it at 3GB used before the reboot.
Updated by pjessen over 2 years ago
pjessen wrote:
I won't pretend to understand much of this stuff, but in the logs I see this, beginning 19 October:
ERROR 2021-10-25 12:11:53,461 12586 django.request Internal Server Error: /archives/search
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/xapian_backend.py", line 27, in
import xapian
File "/usr/lib64/python3.6/site-packages/xapian/init.py", line 24, in
from . import _xapian
ImportError: /lib64/librt.so.1: symbol __pthread_attr_copy, version GLIBC_PRIVATE not defined in file libpthread.so.0 with link time reference
The last time this happened was 25 October, since then nothing. Heisenbug?
Updated by pjessen over 2 years ago
bmwiedemann wrote:
I increased RAM of os-mailman3 from 8 to 12GB
Maybe mailman's uwsgi process has a memory leak. I saw it at 3GB used before the reboot.
I was just now keeping an eye on it - it is at 2.8Gb, and taking up an entire CPU. Sometimes it drops down to 2.4G, but then quickly builds up again.
Updated by pjessen over 2 years ago
pjessen wrote:
bmwiedemann wrote:
I increased RAM of os-mailman3 from 8 to 12GB
Maybe mailman's uwsgi process has a memory leak. I saw it at 3GB used before the reboot.
I was just now keeping an eye on it - it is at 2.8Gb, and taking up an entire CPU. Sometimes it drops down to 2.4G, but then quickly builds up again.
Currently 'uwsgi' is at almost 6Gb:
top - 09:41:05 up 7 days, 18:20, 1 user, load average: 0.56, 0.50, 0.45
Tasks: 194 total, 1 running, 193 sleeping, 0 stopped, 0 zombie
%Cpu(s): 15.5 us, 0.9 sy, 0.0 ni, 82.6 id, 0.2 wa, 0.0 hi, 0.6 si, 0.4 st
MiB Mem : 11754.97+total, 1635.348 free, 9605.910 used, 513.719 buff/cache
MiB Swap: 255.996 total, 1.520 free, 254.477 used. 1817.375 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25647 mailman 20 0 6088584 5.611g 14436 S 38.67 48.87 219:24.02 uwsgi
25452 nginx 20 0 1414236 1.245g 2116 S 0.000 10.84 32:53.13 nginx
25456 nginx 20 0 1414236 1.244g 2080 S 0.000 10.83 0:08.64 nginx
25453 nginx 20 0 1414236 1.243g 8 S 0.333 10.82 13:42.09 nginx
25454 nginx 20 0 1414236 1.242g 200 S 0.000 10.82 0:36.08 nginx
25455 nginx 20 0 1414236 1.242g 840 S 0.000 10.82 0:15.16 nginx
25457 nginx 20 0 1414236 1.240g 2080 S 0.000 10.80 0:05.68 nginx
1897 root 20 0 1401984 1.238g 4 S 0.000 10.78 1:49.58 nginx
Updated by pjessen over 2 years ago
For the apparent memory leak in uwsgi, see #102203
Updated by pjessen over 2 years ago
FWIW, the search index still appears to be being updated:
# l /var/lib/mailman_webui/xapian_index
total 133482760
drwxr-xr-x 2 mailman mailman 126 Dec 2 08:00 ./
drwxr-xr-x 6 mailman mailman 4096 Apr 21 2021 ../
-rw-r--r-- 1 mailman mailman 5250809856 Dec 2 08:00 docdata.glass
-rw-r--r-- 1 mailman mailman 0 Dec 2 08:00 flintlock
-rw-r--r-- 1 mailman mailman 183 Dec 2 08:00 iamglass
-rw-r--r-- 1 mailman mailman 85624954880 Dec 1 03:10 position.glass
-rw-r--r-- 1 mailman mailman 32488120320 Dec 2 08:00 postlist.glass
-rw-r--r-- 1 mailman mailman 13322428416 Dec 2 08:00 termlist.glass
However, searching on recent posts on e.g. factory does not show any recent results.
Updated by pjessen over 2 years ago
The search index still appears to be being updated:
l /var/lib/mailman_webui/xapian_index
total 133482760
drwxr-xr-x 2 mailman mailman 126 Jan 18 10:00 ./
drwxr-xr-x 6 mailman mailman 4096 Dec 16 14:35 ../
-rw-r--r-- 1 mailman mailman 5250809856 Jan 18 10:00 docdata.glass
-rw-r--r-- 1 mailman mailman 0 Jan 18 10:00 flintlock
-rw-r--r-- 1 mailman mailman 182 Jan 18 10:00 iamglass
-rw-r--r-- 1 mailman mailman 85624954880 Dec 14 14:46 position.glass
-rw-r--r-- 1 mailman mailman 32488120320 Jan 18 10:00 postlist.glass
-rw-r--r-- 1 mailman mailman 13322428416 Jan 18 10:00 termlist.glass
Updated by pjessen about 2 years ago
- Assignee deleted (
pjessen)
As far as I can tell, the list search facility remains broken. I'm un-assigning myself, I simply have no idea.
Updated by DocB over 1 year ago
Hi, I just stumbled again over the broken search on the mailing list archives.
I understand Per (thanks for your effort!) has no further idea - how do we proceed? Asked in oS mailinglists?