Project

General

Profile

tickets #94381

mailman3: need the root filesystem doubled

Added by pjessen about 1 month ago. Updated 16 days ago.

Status:
Closed
Priority:
Normal
Assignee:
opensuse-admin
Category:
Core services and infra
Target version:
-
Start date:
2021-06-21
Due date:
% Done:

100%

Estimated time:

Description

The root filesystem on mailman3 keeps running out of space, can we please have it doubled in size?

History

#1 Updated by pjessen about 1 month ago

  • Private changed from Yes to No

#2 Updated by lrupp about 1 month ago

  • Status changed from New to Workable

pjessen wrote:

The root filesystem on mailman3 keeps running out of space, can we please have it doubled in size?

We can. But before we just simply try to circumvent the symtomps, I like to know if we shouldn't check what exactly is filling it up.

  1. The current /var/log/nginx/error.log is full of:

    client intended to send too large body: 2134614 bytes, client: 127.0.0.1, server: lists.opensuse.org, request: "POST /archives/api/mailman/archive?key=dC762xPVifbHtF4hRYSu45Xp HTTP/1.1", host: "localhost"
    

    IMHO this should be checked and fixed.

  2. /var/lib/mailman (more detail: /var/lib/mailman/messages/) needs 1.6G size. Does that make sense? - It looks to me like mailman is storing the mails in some crippled format inside these directories (instead of doing it like mlmmj in normal postfix format aka plain-text). Note: I'm fine with that, but in such a case we should think about moving this into a dedicated partition or separate drive.

  3. If I run

    du -sxch * | sort -h
    

    on '/', I get a total of 5.2G used. But df -h reports 8.7G used. I guess the difference of 3.5G is hidden below /var/lib/mailman_webui/xapian_index, which is now on a dedicated disk.


Suggestion:

  1. Additional disk with 20G for /var/lib/mailman/
  2. Short downtime for mailman to
    • see if /var/lib/mailman_webui/xapian_index on vda2 still contains data
    • sync /var/lib/mailman/ do the new disk
  3. someone will look after the error from nginx.

I could do 1&2 for you, but 3 is something for a mailman expert.

Pro:

  • /var/lib/mailman_webui/xapian_index and /var/lib/mailman/ will be on dedicated disks, allowing to attach them over to another machine quickly
  • / will mainly just contain logs and very little other moving data and can stay "as is"

Con:

  • short downtime needed for the move
  • unknown at the moment, if there might be other resource hungry stuff popping up. Worst case, we need to do a similar exercise again.

What do you think?
Other ideas?

Regards,
Lars

#3 Updated by pjessen about 1 month ago

lrupp wrote:

pjessen wrote:

The root filesystem on mailman3 keeps running out of space, can we please have it doubled in size?

We can. But before we just simply try to circumvent the symtomps, I like to know if we shouldn't check what exactly is filling it up.

Thank for the fast feedback, Lars.

We definitely should find out what's going on, but it might be easier if we fix the symptoms first :-)

  1. /var/lib/mailman (more detail: /var/lib/mailman/messages/) needs 1.6G size. Does that make sense? - It looks to me like mailman is storing the mails in some crippled format inside these directories (instead of doing it like mlmmj in normal postfix format aka plain-text).

Mailman does things in weird and wonderful ways - I think those messages might be those that have been held back for moderation, typically either spam or from non-members. Currently 14935. I'm deleting some now (via the web interface), but they don't immediately disappear, maybe there is a regular tidy-up routine.

  1. If I run ~~~ du -sxch * | sort -h ~~~ on '/', I get a total of 5.2G used. But df -h reports 8.7G used. I guess the difference of 3.5G is hidden below /var/lib/mailman_webui/xapian_index, which is now on a dedicated disk.

Oh, that's not good.

Suggestion:

  1. Additional disk with 20G for /var/lib/mailman/
  2. Short downtime for mailman to
    • see if /var/lib/mailman_webui/xapian_index on vda2 still contains data
    • sync /var/lib/mailman/ do the new disk

1) /var/lib/mailman is currently only 1.6Gb, of which /var/lib/mailman/messages is 1.2Gb. I would hope those will disappear as we look through messages held for moderation.
2) good idea, I'll do that now.

#4 Updated by pjessen about 1 month ago

pjessen wrote:

lrupp wrote:

  1. If I run ~~~ du -sxch * | sort -h ~~~ on '/', I get a total of 5.2G used. But df -h reports 8.7G used. I guess the difference of 3.5G is hidden below /var/lib/mailman_webui/xapian_index, which is now on a dedicated disk.

Oh, that's not good.

Have just stopped mailman, unmounted and checked -

mailman3 (lists.o.o):/var/log # umount /var/lib/mailman_webui/xapian_index
mailman3 (lists.o.o):/var/log # l /var/lib/mailman_webui/xapian_index
total 8
drwxr-xr-x 2 mailman mailman 4096 Mar  9 15:37 ./
drwxr-xr-x 6 mailman mailman 4096 Apr 21 12:34 ../

#5 Updated by pjessen about 1 month ago

I don't understand why we should have such a discrepancy between what 'du' and 'df' reports - 3.5Gb is a third of the filesystem.

#6 Updated by pjessen about 1 month ago

FWIW, I don't know what is being stored in /var/lib/mailman/messages - some of the messages are definitely pending review by a moderator, but some of them are very old and don't show as pending at all. I have discarded maybe a thousand messages today, does not seem to affect /var/lib/mailman/messages at all.

#7 Updated by pjessen about 1 month ago

pjessen wrote:

FWIW, I don't know what is being stored in /var/lib/mailman/messages - some of the messages are definitely pending review by a moderator, but some of them are very old and don't show as pending at all. I have discarded maybe a thousand messages today, does not seem to affect /var/lib/mailman/messages at all.

I forgot I'd opened issue #90653 on the above, back in April.

#8 Updated by lrupp about 1 month ago

  • % Done changed from 0 to 30

FYI:

 /dev/vdc         20G  1.7G   19G   9% /data

Has a first sync of /var/lib/mailman/. /etc/fstab also contains the correct entry, but is currently commented out.

Regards,
Lars

#9 Updated by pjessen about 1 month ago

Yeah, I saw that /dev/vdc the other day :-)
I don't know if it is worth doing much about /var/lib/mailman - the only thing that is growing is /var/lib/mailman/messages, and that seems to be due to a lack of deleting messages. Sasi suggested maybe a missing cronjob. See #90653. When we get that going, /var/lib/mailman/ will only have maybe 400-500Mb.

#10 Updated by lrupp 16 days ago

  • Status changed from Workable to Closed
  • % Done changed from 30 to 100

As the initial request is done, let's close this.

Also available in: Atom PDF