tickets #129463
closedmailman3 - the admin-auto list has 10000 held messages
100%
Description
Not really a ticket - more of a "aaaaaaaaaaaarg!!!!".
At 3 minutes and 20 seconds per discard, that will take about three weeks, full-time. and that is after automating it.
Updated by pjessen over 1 year ago
- Due date set to 2023-05-24
- Category set to Mailing lists
- Assignee set to pjessen
- Private changed from Yes to No
I have reconfigured mailman to have 6 rest-api workers, and I am doing 4 discards in parallel. I guess the mailman machine might be running a bit hot :-)
Updated by pjessen over 1 year ago
- Due date changed from 2023-05-24 to 2023-05-23
- Status changed from New to In Progress
- % Done changed from 0 to 20
Hmm, this isn't going too badly. Sofar 1900 discarded postings, that suggests it could be done in five days.
Updated by luc14n0 over 1 year ago
Let me take this opportunity here to ask a question: What exactly is admin-auto?
The name suggests it's automating something, of course, but I can't deduce what exactly. And is it normal for it to held messages (like that)?
Updated by pjessen over 1 year ago
luc14n0 wrote:
Let me take this opportunity here to ask a question: What exactly is admin-auto?
It is the standard destination for all kinds of sysadmin output from openSUSE machines. You can subscribe to the list, there are all kinds of amazing information :-)
For instance, I have a cronjob on mailman3 that tidies up the postfix queue, to get rid of undeliverable messages. The output goes to admin-auto:
postfix queue clean-up: mails from long (22 chars or more) unresolvable domains:
postsuper: B85B370D: removed
postsuper: 7BA24FA: removed
postsuper: D80EB4470: removed
postsuper: D984984E: removed
postsuper: Deleted: 4 messages
The name suggests it's automating something, of course, but I can't deduce what exactly. And is it normal for it to held messages (like that)?
They were all held due to "Implicit destination" and they were all sent by postfix on mailman3 - unable to deliver to anna, due to insufficient storage. They ought to have been just accepted, although that would only have exacerbated the situation :-)
Updated by pjessen over 1 year ago
- % Done changed from 20 to 40
pjessen wrote:
Hmm, this isn't going too badly. Sofar 1900 discarded postings, that suggests it could be done in five days.
Currently 4002 messages discarded, so approx 2000 per day, that's not too bad.
Updated by pjessen over 1 year ago
The discarding of held messages finished last night. It got significantly faster as it progressed.
May 17 - 1013 discarded (not a full day)
May 18 - 2113 discarded.
May 19 - 2823 discarded
May 20 - 4631 discarded
Total - 10584.
From more than 3 minutes to complete, it has dropped to around 30 seconds. This seems to suggest that the number of messages held has a serious impact on performance.
There are a few other lists with hundreds of held messages, I'm going to tidy up those too, then we'll see.
Updated by pjessen over 1 year ago
pjessen wrote:
There are a few other lists with hundreds of held messages, I'm going to tidy up those too, then we'll see.
611 press.lists.opensuse.org 551 commit.lists.opensuse.org 445 security-announce.lists.opensuse.org 131 security.lists.opensuse.org 110 conference.lists.opensuse.org 109 admin.lists.opensuse.org 74 marketing.lists.opensuse.org 56 shop.lists.opensuse.org 43 mirror.lists.opensuse.org 38 updates.lists.opensuse.org 29 zypp-commit.lists.opensuse.org 27 summit.lists.opensuse.org 27 news.lists.opensuse.org 15 forums-admin.lists.opensuse.org 10 sourcedvd.lists.opensuse.org 9 wicked-commit.lists.opensuse.org 9 obs-tests.lists.opensuse.org 7 kernel-bugs.lists.opensuse.org 6 support.lists.opensuse.org 6 announce.lists.opensuse.org 5 weblate-admin.lists.opensuse.org 4 obs-devel.lists.opensuse.org 3 membership-officials.lists.opensuse.org 3 factory.lists.opensuse.org 2 yast-devel.lists.opensuse.org 2 tweet-master.lists.opensuse.org 2 project.lists.opensuse.org 2 cfp.lists.opensuse.org 2 arm.lists.opensuse.org 1 web.lists.opensuse.org 1 users.lists.opensuse.org 1 obs-commits.lists.opensuse.org 1 maintenance.lists.opensuse.org 1 kernel.lists.opensuse.org 1 ircops.lists.opensuse.org
Updated by pjessen over 1 year ago
Well, I cleared out quite a bit of it, I discarded all held messages for press@lists for instance. It seems that one is also ready filling up again.
370 security-announce.lists.opensuse.org
367 commit.lists.opensuse.org
92 press.lists.opensuse.org
73 conference.lists.opensuse.org
54 marketing.lists.opensuse.org
40 kernel.lists.opensuse.org
33 mirror.lists.opensuse.org
22 news.lists.opensuse.org
15 forums-admin.lists.opensuse.org
10 sourcedvd.lists.opensuse.org
9 obs-tests.lists.opensuse.org
6 support.lists.opensuse.org
6 kernel-bugs.lists.opensuse.org
6 announce.lists.opensuse.org
4 weblate-admin.lists.opensuse.org
4 obs-devel.lists.opensuse.org
3 wicked-commit.lists.opensuse.org
3 membership-officials.lists.opensuse.org
3 factory.lists.opensuse.org
3 admin.lists.opensuse.org
2 yast-devel.lists.opensuse.org
2 cfp.lists.opensuse.org
2 arm.lists.opensuse.org
1 users.lists.opensuse.org
1 summit.lists.opensuse.org
1 obs-commits.lists.opensuse.org
1 maintenance.lists.opensuse.org
Updated by pjessen over 1 year ago
- Status changed from In Progress to Resolved
- % Done changed from 40 to 100
I'm closing this as resolved - messages can now be discarded relatively quickly, even in batches - e.g. 5 in 18seconds. The conclusion has to be that having too many held messages is a problem. Makes me wonder if the same applies to having too many 'non-members'.
Updated by pjessen over 1 year ago
- Related to tickets #121993: unable to process held_messages (reject, discard) in mailman added
Updated by pjessen over 1 year ago
- Related to tickets #101848: Poor performance of lists.opensuse.org - web ui, running very slowly. added