Project

General

Profile

Actions

tickets #129463

closed

mailman3 - the admin-auto list has 10000 held messages

Added by pjessen 12 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Mailing lists
Target version:
-
Start date:
2023-05-17
Due date:
2023-05-23
% Done:

100%

Estimated time:

Description

Not really a ticket - more of a "aaaaaaaaaaaarg!!!!".

At 3 minutes and 20 seconds per discard, that will take about three weeks, full-time. and that is after automating it.


Related issues 2 (0 open2 closed)

Related to openSUSE admin - tickets #121993: unable to process held_messages (reject, discard) in mailmanResolvedpjessen2022-12-142022-12-22

Actions
Related to openSUSE admin - tickets #101848: Poor performance of lists.opensuse.org - web ui, running very slowly.Resolvedpjessen2021-11-02

Actions
Actions #1

Updated by pjessen 12 months ago

  • Due date set to 2023-05-24
  • Category set to Mailing lists
  • Assignee set to pjessen
  • Private changed from Yes to No

I have reconfigured mailman to have 6 rest-api workers, and I am doing 4 discards in parallel. I guess the mailman machine might be running a bit hot :-)

Actions #2

Updated by pjessen 12 months ago

  • Due date changed from 2023-05-24 to 2023-05-23
  • Status changed from New to In Progress
  • % Done changed from 0 to 20

Hmm, this isn't going too badly. Sofar 1900 discarded postings, that suggests it could be done in five days.

Actions #3

Updated by luc14n0 12 months ago

Let me take this opportunity here to ask a question: What exactly is admin-auto?

The name suggests it's automating something, of course, but I can't deduce what exactly. And is it normal for it to held messages (like that)?

Actions #4

Updated by pjessen 11 months ago

luc14n0 wrote:

Let me take this opportunity here to ask a question: What exactly is admin-auto?

It is the standard destination for all kinds of sysadmin output from openSUSE machines. You can subscribe to the list, there are all kinds of amazing information :-)
For instance, I have a cronjob on mailman3 that tidies up the postfix queue, to get rid of undeliverable messages. The output goes to admin-auto:

postfix queue clean-up: mails from long (22 chars or more) unresolvable domains:
postsuper: B85B370D: removed
postsuper: 7BA24FA: removed
postsuper: D80EB4470: removed
postsuper: D984984E: removed
postsuper: Deleted: 4 messages

The name suggests it's automating something, of course, but I can't deduce what exactly. And is it normal for it to held messages (like that)?

They were all held due to "Implicit destination" and they were all sent by postfix on mailman3 - unable to deliver to anna, due to insufficient storage. They ought to have been just accepted, although that would only have exacerbated the situation :-)

Actions #5

Updated by pjessen 11 months ago

  • % Done changed from 20 to 40

pjessen wrote:

Hmm, this isn't going too badly. Sofar 1900 discarded postings, that suggests it could be done in five days.

Currently 4002 messages discarded, so approx 2000 per day, that's not too bad.

Actions #6

Updated by pjessen 11 months ago

The discarding of held messages finished last night. It got significantly faster as it progressed.
May 17 - 1013 discarded (not a full day)
May 18 - 2113 discarded.
May 19 - 2823 discarded
May 20 - 4631 discarded
Total - 10584.
From more than 3 minutes to complete, it has dropped to around 30 seconds. This seems to suggest that the number of messages held has a serious impact on performance.
There are a few other lists with hundreds of held messages, I'm going to tidy up those too, then we'll see.

Actions #7

Updated by pjessen 11 months ago

pjessen wrote:

There are a few other lists with hundreds of held messages, I'm going to tidy up those too, then we'll see.

    611 press.lists.opensuse.org
    551 commit.lists.opensuse.org
    445 security-announce.lists.opensuse.org
    131 security.lists.opensuse.org
    110 conference.lists.opensuse.org
    109 admin.lists.opensuse.org
     74 marketing.lists.opensuse.org
     56 shop.lists.opensuse.org
     43 mirror.lists.opensuse.org
     38 updates.lists.opensuse.org
     29 zypp-commit.lists.opensuse.org
     27 summit.lists.opensuse.org
     27 news.lists.opensuse.org
     15 forums-admin.lists.opensuse.org
     10 sourcedvd.lists.opensuse.org
      9 wicked-commit.lists.opensuse.org
      9 obs-tests.lists.opensuse.org
      7 kernel-bugs.lists.opensuse.org
      6 support.lists.opensuse.org
      6 announce.lists.opensuse.org
      5 weblate-admin.lists.opensuse.org
      4 obs-devel.lists.opensuse.org
      3 membership-officials.lists.opensuse.org
      3 factory.lists.opensuse.org
      2 yast-devel.lists.opensuse.org
      2 tweet-master.lists.opensuse.org
      2 project.lists.opensuse.org
      2 cfp.lists.opensuse.org
      2 arm.lists.opensuse.org
      1 web.lists.opensuse.org
      1 users.lists.opensuse.org
      1 obs-commits.lists.opensuse.org
      1 maintenance.lists.opensuse.org
      1 kernel.lists.opensuse.org
      1 ircops.lists.opensuse.org
Actions #8

Updated by pjessen 11 months ago

Well, I cleared out quite a bit of it, I discarded all held messages for press@lists for instance. It seems that one is also ready filling up again.

    370 security-announce.lists.opensuse.org
    367 commit.lists.opensuse.org
     92 press.lists.opensuse.org
     73 conference.lists.opensuse.org
     54 marketing.lists.opensuse.org
     40 kernel.lists.opensuse.org
     33 mirror.lists.opensuse.org
     22 news.lists.opensuse.org
     15 forums-admin.lists.opensuse.org
     10 sourcedvd.lists.opensuse.org
      9 obs-tests.lists.opensuse.org
      6 support.lists.opensuse.org
      6 kernel-bugs.lists.opensuse.org
      6 announce.lists.opensuse.org
      4 weblate-admin.lists.opensuse.org
      4 obs-devel.lists.opensuse.org
      3 wicked-commit.lists.opensuse.org
      3 membership-officials.lists.opensuse.org
      3 factory.lists.opensuse.org
      3 admin.lists.opensuse.org
      2 yast-devel.lists.opensuse.org
      2 cfp.lists.opensuse.org
      2 arm.lists.opensuse.org
      1 users.lists.opensuse.org
      1 summit.lists.opensuse.org
      1 obs-commits.lists.opensuse.org
      1 maintenance.lists.opensuse.org
Actions #9

Updated by pjessen 11 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 40 to 100

I'm closing this as resolved - messages can now be discarded relatively quickly, even in batches - e.g. 5 in 18seconds. The conclusion has to be that having too many held messages is a problem. Makes me wonder if the same applies to having too many 'non-members'.

Actions #10

Updated by pjessen 11 months ago

  • Related to tickets #121993: unable to process held_messages (reject, discard) in mailman added
Actions #11

Updated by pjessen 10 months ago

  • Related to tickets #101848: Poor performance of lists.opensuse.org - web ui, running very slowly. added
Actions

Also available in: Atom PDF