tickets #129463
closedmailman3 - the admin-auto list has 10000 held messages
100%
Description
Not really a ticket - more of a "aaaaaaaaaaaarg!!!!".
At 3 minutes and 20 seconds per discard, that will take about three weeks, full-time. and that is after automating it.
Updated by pjessen about 2 years ago
- Due date set to 2023-05-24
- Category set to Mailing lists
- Assignee set to pjessen
- Private changed from Yes to No
I have reconfigured mailman to have 6 rest-api workers, and I am doing 4 discards in parallel. I guess the mailman machine might be running a bit hot :-)
Updated by pjessen about 2 years ago
- Due date changed from 2023-05-24 to 2023-05-23
- Status changed from New to In Progress
- % Done changed from 0 to 20
Hmm, this isn't going too badly. Sofar 1900 discarded postings, that suggests it could be done in five days.
Updated by luc14n0 about 2 years ago
Let me take this opportunity here to ask a question: What exactly is admin-auto?
The name suggests it's automating something, of course, but I can't deduce what exactly. And is it normal for it to held messages (like that)?
Updated by pjessen about 2 years ago
luc14n0 wrote:
Let me take this opportunity here to ask a question: What exactly is admin-auto?
It is the standard destination for all kinds of sysadmin output from openSUSE machines. You can subscribe to the list, there are all kinds of amazing information :-)
For instance, I have a cronjob on mailman3 that tidies up the postfix queue, to get rid of undeliverable messages. The output goes to admin-auto:
postfix queue clean-up: mails from long (22 chars or more) unresolvable domains:
postsuper: B85B370D: removed
postsuper: 7BA24FA: removed
postsuper: D80EB4470: removed
postsuper: D984984E: removed
postsuper: Deleted: 4 messages
The name suggests it's automating something, of course, but I can't deduce what exactly. And is it normal for it to held messages (like that)?
They were all held due to "Implicit destination" and they were all sent by postfix on mailman3 - unable to deliver to anna, due to insufficient storage. They ought to have been just accepted, although that would only have exacerbated the situation :-)
Updated by pjessen about 2 years ago
- % Done changed from 20 to 40
pjessen wrote:
Hmm, this isn't going too badly. Sofar 1900 discarded postings, that suggests it could be done in five days.
Currently 4002 messages discarded, so approx 2000 per day, that's not too bad.
Updated by pjessen about 2 years ago
The discarding of held messages finished last night. It got significantly faster as it progressed.
May 17 - 1013 discarded (not a full day)
May 18 - 2113 discarded.
May 19 - 2823 discarded
May 20 - 4631 discarded
Total - 10584.
From more than 3 minutes to complete, it has dropped to around 30 seconds. This seems to suggest that the number of messages held has a serious impact on performance.
There are a few other lists with hundreds of held messages, I'm going to tidy up those too, then we'll see.
Updated by pjessen about 2 years ago
pjessen wrote:
There are a few other lists with hundreds of held messages, I'm going to tidy up those too, then we'll see.
611 press.lists.opensuse.org
551 commit.lists.opensuse.org
445 security-announce.lists.opensuse.org
131 security.lists.opensuse.org
110 conference.lists.opensuse.org
109 admin.lists.opensuse.org
74 marketing.lists.opensuse.org
56 shop.lists.opensuse.org
43 mirror.lists.opensuse.org
38 updates.lists.opensuse.org
29 zypp-commit.lists.opensuse.org
27 summit.lists.opensuse.org
27 news.lists.opensuse.org
15 forums-admin.lists.opensuse.org
10 sourcedvd.lists.opensuse.org
9 wicked-commit.lists.opensuse.org
9 obs-tests.lists.opensuse.org
7 kernel-bugs.lists.opensuse.org
6 support.lists.opensuse.org
6 announce.lists.opensuse.org
5 weblate-admin.lists.opensuse.org
4 obs-devel.lists.opensuse.org
3 membership-officials.lists.opensuse.org
3 factory.lists.opensuse.org
2 yast-devel.lists.opensuse.org
2 tweet-master.lists.opensuse.org
2 project.lists.opensuse.org
2 cfp.lists.opensuse.org
2 arm.lists.opensuse.org
1 web.lists.opensuse.org
1 users.lists.opensuse.org
1 obs-commits.lists.opensuse.org
1 maintenance.lists.opensuse.org
1 kernel.lists.opensuse.org
1 ircops.lists.opensuse.org
Updated by pjessen about 2 years ago
Well, I cleared out quite a bit of it, I discarded all held messages for press@lists for instance. It seems that one is also ready filling up again.
370 security-announce.lists.opensuse.org
367 commit.lists.opensuse.org
92 press.lists.opensuse.org
73 conference.lists.opensuse.org
54 marketing.lists.opensuse.org
40 kernel.lists.opensuse.org
33 mirror.lists.opensuse.org
22 news.lists.opensuse.org
15 forums-admin.lists.opensuse.org
10 sourcedvd.lists.opensuse.org
9 obs-tests.lists.opensuse.org
6 support.lists.opensuse.org
6 kernel-bugs.lists.opensuse.org
6 announce.lists.opensuse.org
4 weblate-admin.lists.opensuse.org
4 obs-devel.lists.opensuse.org
3 wicked-commit.lists.opensuse.org
3 membership-officials.lists.opensuse.org
3 factory.lists.opensuse.org
3 admin.lists.opensuse.org
2 yast-devel.lists.opensuse.org
2 cfp.lists.opensuse.org
2 arm.lists.opensuse.org
1 users.lists.opensuse.org
1 summit.lists.opensuse.org
1 obs-commits.lists.opensuse.org
1 maintenance.lists.opensuse.org
Updated by pjessen about 2 years ago
- Status changed from In Progress to Resolved
- % Done changed from 40 to 100
I'm closing this as resolved - messages can now be discarded relatively quickly, even in batches - e.g. 5 in 18seconds. The conclusion has to be that having too many held messages is a problem. Makes me wonder if the same applies to having too many 'non-members'.
Updated by pjessen about 2 years ago
- Related to tickets #121993: unable to process held_messages (reject, discard) in mailman added
Updated by pjessen almost 2 years ago
- Related to tickets #101848: Poor performance of lists.opensuse.org - web ui, running very slowly. added