Project

General

Profile

Actions

tickets #138290

closed

anna/elsa - which smtp_bind_address to use?

Added by pjessen 7 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
2023-10-19
Due date:
% Done:

0%

Estimated time:

Description

I'm opening a new ticket instead of pursuing this under #135779.

A quick check on progressoo reveals that mails continue to be bounced, some 579 in the current log:

2023-10-18T19:30:24.178784+00:00 progressoo postfix/pipe[4675]: E98C5685B: to=<redmine-opensuse-admin+admin@localhost.redmine>, orig_to=<redmine+admin@progressoo.infra.opensuse.org>, relay=redmineprivate, delay=0.22, delays=0.01/0/0/0.22, dsn=5.7.0, status=bounced (permission denied. Command output: Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information. )
2023-10-19T02:47:10.379018+00:00 progressoo postfix/pipe[21070]: 27846685B: to=<redmine-opensuse-admin+admin@localhost.redmine>, orig_to=<redmine+admin@progressoo.infra.opensuse.org>, relay=redmineprivate, delay=0.22, delays=0.01/0/0/0.21, dsn=5.7.0, status=bounced (permission denied. Command output: Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information. )
2023-10-19T05:00:21.703290+00:00 progressoo postfix/pipe[3420]: 71530685B: to=<redmine-opensuse-admin+admin@localhost.redmine>, orig_to=<redmine+admin@progressoo.infra.opensuse.org>, relay=redmineprivate, delay=0.25, delays=0.01/0.01/0/0.22, dsn=5.7.0, status=bounced (permission denied. Command output: Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information. )

The most recent one, 71530685B, seems to be an NDR sent from anna.

2023-10-19T05:00:21.464471+00:00 progressoo postfix/smtpd[3416]: 71530685B: client=anna.infra.opensuse.org[192.168.47.102]
2023-10-19T05:00:21.465655+00:00 progressoo postfix/cleanup[3419]: 71530685B: message-id=<20231019050019.860CF243E4@anna.opensuse.org>
2023-10-19T05:00:21.469400+00:00 progressoo postfix/qmgr[9233]: 71530685B: from=<>, size=5038, nrcpt=1 (queue active)
2023-10-19T05:00:21.703290+00:00 progressoo postfix/pipe[3420]: 71530685B: to=<redmine-opensuse-admin+admin@localhost.redmine>, orig_to=<redmine+admin@progressoo.infra.opensuse.org>, relay=redmineprivate, delay=0.25, delays=0.01/0.01/0/0.22, dsn=5.7.0, status=bounced (permission denied. Command output: Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information. )
2023-10-19T05:00:21.704097+00:00 progressoo postfix/qmgr[9233]: 71530685B: removed

This came from mx1:

2023-10-19T05:00:21.408488+00:00 anna postfix/smtpd[19274]: 63A7E2436B: client=mx1.infra.opensuse.org[192.168.47.95]
2023-10-19T05:00:21.409372+00:00 anna postfix/cleanup[19559]: 63A7E2436B: message-id=<20231019050019.860CF243E4@anna.opensuse.org>
2023-10-19T05:00:21.415853+00:00 anna postfix/qmgr[26767]: 63A7E2436B: from=<>, size=4801, nrcpt=1 (queue active)
2023-10-19T05:00:21.470047+00:00 anna postfix/smtp[19853]: 63A7E2436B: to=<redmine+admin@progressoo.infra.opensuse.org>, relay=progressoo.infra.opensuse.org[192.168.47.34]:25, delay=0.06, delays=0.01/0/0.03/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 71530685B)
2023-10-19T05:00:21.470352+00:00 anna postfix/qmgr[26767]: 63A7E2436B: removed

which in turn came from static.opensuse.org:

2023-10-19T05:00:19.588927+00:00 mx1 postfix/smtpd[21044]: connect from static.opensuse.org[2001:67c:2178:8::18]
2023-10-19T05:00:19.600557+00:00 mx1 postfix/smtpd[21044]: Anonymous TLS connection established from static.opensuse.org[2001:67c:2178:8::18]: TLSv1.3 with cipher TLS_AES_256_
GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384
2023-10-19T05:00:19.602059+00:00 mx1 postfix/smtpd[21044]: NOQUEUE: client=static.opensuse.org[2001:67c:2178:8::18]
2023-10-19T05:00:19.634690+00:00 mx1 postfix/smtpd[22134]: connect from localhost[127.0.0.1]
2023-10-19T05:00:19.647420+00:00 mx1 postfix/smtpd[22134]: 9DFDF159: client=localhost[127.0.0.1], orig_client=static.opensuse.org[2001:67c:2178:8::18]
2023-10-19T05:00:19.649273+00:00 mx1 postsrsd[22137]: srs_forward: <""> not rewritten: No at sign in sender address
2023-10-19T05:00:19.651552+00:00 mx1 spampd[16737]: processing message <20231019050019.860CF243E4@anna.opensuse.org> for <redmine@opensuse.org> ORCPT=rfc822;redmine@opensuse.org
2023-10-19T05:00:21.380708+00:00 mx1 spampd[16737]: clean message <20231019050019.860CF243E4@anna.opensuse.org> (-2.20/5.01) from <> for <redmine@opensuse.org> ORCPT=rfc822;redmine@opensuse.org in 1.73s, 4123 bytes.
2023-10-19T05:00:21.384297+00:00 mx1 postfix/cleanup[22136]: 9DFDF159: message-id=<20231019050019.860CF243E4@anna.opensuse.org>
2023-10-19T05:00:21.387913+00:00 mx1 postfix/qmgr[31947]: 9DFDF159: from=<>, size=4572, nrcpt=1 (queue active)
2023-10-19T05:00:21.388388+00:00 mx1 postfix/smtpd[21044]: proxy-accept: END-OF-MESSAGE: 250 2.0.0 Ok: queued as 9DFDF159; from=<> to=<redmine@opensuse.org> proto=ESMTP helo=<anna.opensuse.org>
2023-10-19T05:00:21.388560+00:00 mx1 postfix/smtpd[21044]: disconnect from static.opensuse.org[2001:67c:2178:8::18] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7
2023-10-19T05:00:21.388885+00:00 mx1 postfix/smtpd[22134]: disconnect from localhost[127.0.0.1] ehlo=1 xforward=1 mail=1 rcpt=1 data=1 quit=1 commands=6
2023-10-19T05:00:21.417098+00:00 mx1 postfix/smtp[22143]: 9DFDF159: to=<redmine+admin@progressoo.infra.opensuse.org>, orig_to=<redmine@opensuse.org>, relay=relay.infra.opensuse.org[192.168.47.4]:25, delay=1.8, delays=1.8/0.02/0/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 63A7E2436B)

It is worth nothing that static.o.o is delivering an NDR with a messageid originating on anna.i.o.o. I wonder where this is going to go.


Related issues 1 (0 open1 closed)

Related to openSUSE admin - tickets #124793: permission denied. Command output: Request was denied by your Redmine server. Resolvedcrameleon2023-02-20

Actions
Actions #1

Updated by pjessen 7 months ago

  • Subject changed from progressoo - mail bounces to progressoo - need proper envelope bounce address for redmine
  • Private changed from Yes to No

Ignoring the confusion by anna.o.o pretending to be static.o.o, I think I have found the issue:

Some or other ticket is updated in redmine, which causes an update to distributed, one goes to riwilliams@suse.com who seems to have left the company:

2023-10-19T05:00:19.532032+00:00 anna postfix/qmgr[26767]: 804AA243CF: from=<redmine@opensuse.org>, size=1200, nrcpt=1 (queue active)
2023-10-19T05:00:19.542389+00:00 anna postfix/smtp[19844]: 524E324383: to=<riwilliams@suse.com>, relay=suse-com.mail.protection.outlook.com[52.101.73.8]:25, delay=0.2, delays=
0.01/0.01/0.11/0.07, dsn=5.4.1, status=bounced (host suse-com.mail.protection.outlook.com[52.101.73.8] said: 550 5.4.1 Recipient address rejected: Access denied. [AM4PEPF00027
A6A.eurprd04.prod.outlook.com 2023-10-19T05:00:19.488Z 08DBCEAE6EA86678] (in reply to RCPT TO command))
2023-10-19T05:00:19.551254+00:00 anna postfix/cleanup[19559]: 860CF243E4: message-id=<20231019050019.860CF243E4@anna.opensuse.org>
2023-10-19T05:00:19.559610+00:00 anna postfix/smtp[19845]: Untrusted TLS connection established to suse-com.mail.protection.outlook.com[52.101.73.12]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
2023-10-19T05:00:19.561973+00:00 anna postfix/bounce[19848]: 524E324383: sender non-delivery notification: 860CF243E4
2023-10-19T05:00:19.564138+00:00 anna postfix/qmgr[26767]: 860CF243E4: from=<>, size=3743, nrcpt=1 (queue active)

The NDR eventually ends up on progresoo, where it is rejected by redmine:

2023-10-19T05:00:21.703290+00:00 progressoo postfix/pipe[3420]: 71530685B: to=<redmine-opensuse-admin+admin@localhost.redmine>, orig_to=<redmine+admin@progressoo.infra.opensuse.org>, relay=redmineprivate, delay=0.25, delays=0.01/0.01/0/0.22, dsn=5.7.0, status=bounced (permission denied. Command output: Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information. )

I'm going to check on those 579 other bounces, it seems like quite a lot, but I think the issue here is -

redmine sends out notifications etc with envelope address redmine@opensuse.org - that probably ought to be improved on such that bounces can be recognised as such.

Actions #2

Updated by pjessen 7 months ago

pjessen wrote in #note-1:

I'm going to check on those 579 other bounces, it seems like quite a lot, but I think the issue here is -

Those 579 are all bounces going back to 2021-04-27. The mail log on progressoo never grows big enough to be rotated. Counting only October 2023, there has been 144 bounces.

Actions #3

Updated by pjessen 7 months ago

pjessen wrote in #note-2:

Counting only October 2023, there has been 144 bounces.

The usual bounce reasons -

Does redmine have some built-in mechanism for dealing with this sort of thing, e.g. in the style of mailman?

Actions #4

Updated by luc14n0 7 months ago

pjessen wrote in #note-2:

pjessen wrote in #note-1:

I'm going to check on those 579 other bounces, it seems like quite a lot, but I think the issue here is -

Those 579 are all bounces going back to 2021-04-27. The mail log on progressoo never grows big enough to be rotated. Counting only October 2023, there has been 144 bounces.

That sure is a long ping-pong match 😁

Actions #5

Updated by crameleon 7 months ago

redmine sends out notifications etc with envelope address redmine@opensuse.org - that probably ought to be improved

We can change the From (envelope?) address, but it's referenced in several places (mostly Postfix rewrite maps on multiple machines), which might lead to regressions.

Does redmine have some built-in mechanism for dealing with this sort of thing

I doubt it. The mail handler is a relatively rudimentary Ruby script which takes an email and forwards it to Redmine using the HTTP API:

https://github.com/redmine/redmine/blob/master/extra/mail_handler/rdm-mailhandler.rb

Actions #6

Updated by crameleon 7 months ago

Those 579 are all bounces going back to 2021-04-27

I do want to point out that until recently progressoo.i.o.o was only running https://progress-test.opensuse.org (now defunct). I repurposed the machine to run the new production setup. Hence such old mail entries are likely not from the currently running Redmine instance.

Actions #7

Updated by crameleon 7 months ago

that probably ought to be improved on such that bounces can be recognised as such.

Would it help to add some code to the Ruby script which just drops bounced emails? If so, is there a good programmatic way to detect such an email based on what data is injected by Postfix?

Edit: I now added -v to the command line which should dump the data to the journal next time.

Actions #8

Updated by pjessen 7 months ago

crameleon wrote in #note-7:

that probably ought to be improved on such that bounces can be recognised as such.

Would it help to add some code to the Ruby script which just drops bounced emails?

If redmine can't process NDRs anyway (I was not aware), we might as well throw them away.
That sounds like something for postfix to do. IOW, "drop anything for redmine that doesn't have a proper sending address". I can do that.

If so, is there a good programmatic way to detect such an email based on what data is injected by Postfix?

Envelope address = <> (empty).

Actions #9

Updated by crameleon 7 months ago

That sounds like something for postfix to do. IOW, "drop anything for redmine that doesn't have a proper sending address". I can do that.

If that's possible in Postfix, that'd be even better - no hacky patching of the script. :-) Let me know what it is you configure, so I can add it to Salt.

Actions #10

Updated by pjessen 7 months ago

  • Status changed from New to In Progress

If that's possible in Postfix, that'd be even better - no hacky patching of the script. :-)

Exactly!

Let me know what it is you configure, so I can add it to Salt.

Willdo. I am just testing a solution, it is a simple sender check.

Actions #11

Updated by pjessen 7 months ago

pjessen wrote in #note-10:

Let me know what it is you configure, so I can add it to Salt.

Willdo. I am just testing a solution, it is a simple sender check.

Old: smtpd_sender_restrictions = lmdb:/etc/postfix/access,reject_unknown_sender_domain
New: smtpd_sender_restrictions = lmdb:/etc/postfix/discard_ndrs,reject_unknown_sender_domain

Contents of /etc/postfix/discard_ndrs:

# poo#138290 discard ndrs etc.
<>      discard

This is a log excerpt:

2023-10-21T10:35:23.271296+00:00 progressoo postfix/smtpd[24532]: connect from mx1.infra.opensuse.org[192.168.47.95]
2023-10-21T10:35:23.281293+00:00 progressoo postfix/smtpd[24532]: NOQUEUE: discard: RCPT from mx1.infra.opensuse.org[192.168.47.95]: <>: Sender address triggers DISCARD action; from=<> to=<redmine+admin@progressoo.infra.opensuse.org> proto=ESMTP helo=<mx1.infra.opensuse.org>
2023-10-21T10:35:23.290794+00:00 progressoo postfix/smtpd[24532]: 46EDF3A92: client=mx1.infra.opensuse.org[192.168.47.95]
2023-10-21T10:35:23.291890+00:00 progressoo postfix/smtpd[24532]: disconnect from mx1.infra.opensuse.org[192.168.47.95] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Actions #12

Updated by pjessen 6 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from pjessen to crameleon

Just in case there was any doubt - this does the job. Just today since midnight, 20 discards.

  • unknown suse.com address (several for the same)
  • bugzilla_noreply@suse.com bounced (several)
  • unknown outlook.de address
  • unknown yahoo.es address
  • relaying denied
  • unknown domain
  • translation-sk@lists.opensuse.org - too many hops ??
  • syscid.com - This message does not meet our delivery requirements

The above may not all have been for redmine, I was checking the log on anna.

@crameleon thanks for offering to saltify.

Actions #13

Updated by crameleon 6 months ago

Thank you for the implementation and the details, Per.

syscid.com - This message does not meet our delivery requirements

That's my server!

Actions #14

Updated by crameleon 6 months ago

I'm not sure it's related to this ticket, but the tracker is very quiet in the last days and two users report having sent emails to admin@o.o which I cannot locate in the queue. Did anything go wrong? The only email related change I performed was removing "infra." from "myhostname" on mx1.

Actions #15

Updated by pjessen 6 months ago

crameleon wrote in #note-14:

I'm not sure it's related to this ticket, but the tracker is very quiet in the last days and two users report having sent emails to admin@o.o which I cannot locate in the queue. Did anything go wrong? The only email related change I performed was removing "infra." from "myhostname" on mx1.

Looks like anna is having trouble talking to progressoo.

2023-10-31T18:48:41.037648+00:00 anna postfix/error[29947]: 05E372151A: to=<redmine+admin@progressoo.infra.opensuse.org>, relay=none, delay=0.01, delays=0.01/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to progressoo.infra.opensuse.org[192.168.47.34]:25: Connection timed out)

Yep, just retried, it is timing out.

Actions #16

Updated by crameleon 6 months ago

The timeout is odd, the attempts from anna do show up:

Oct 31 15:48:30 progressoo postfix/smtpd[1256]: connect from mx1.infra.opensuse.org[192.168.47.95]
Oct 31 15:48:30 progressoo postfix/smtpd[1256]: NOQUEUE: discard: RCPT from mx1.infra.opensuse.org[192.168.47.95]: <>: Sender address triggers DISCARD action; from=<> to=<redmine+admin@progressoo.infra.opensuse.org> proto=ESMTP helo=<mx1.opensuse.org>
Oct 31 15:48:30 progressoo postfix/smtpd[1256]: F121D453A: client=mx1.infra.opensuse.org[192.168.47.95]
Oct 31 15:48:30 progressoo postfix/smtpd[1256]: disconnect from mx1.infra.opensuse.org[192.168.47.95] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Oct 31 18:54:21 progressoo postfix/smtpd[22518]: connect from anna.infra.opensuse.org[192.168.47.102]
Oct 31 18:54:26 progressoo postfix/smtpd[22518]: disconnect from anna.infra.opensuse.org[192.168.47.102] quit=1 commands=1
Actions #17

Updated by pjessen 6 months ago

crameleon wrote in #note-16:

The timeout is odd, the attempts from anna do show up:

Oct 31 15:48:30 progressoo postfix/smtpd[1256]: connect from mx1.infra.opensuse.org[192.168.47.95]
Oct 31 15:48:30 progressoo postfix/smtpd[1256]: NOQUEUE: discard: RCPT from mx1.infra.opensuse.org[192.168.47.95]: <>: Sender address triggers DISCARD action; from=<> to=<redmine+admin@progressoo.infra.opensuse.org> proto=ESMTP helo=<mx1.opensuse.org>
Oct 31 15:48:30 progressoo postfix/smtpd[1256]: F121D453A: client=mx1.infra.opensuse.org[192.168.47.95]
Oct 31 15:48:30 progressoo postfix/smtpd[1256]: disconnect from mx1.infra.opensuse.org[192.168.47.95] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Oct 31 18:54:21 progressoo postfix/smtpd[22518]: connect from anna.infra.opensuse.org[192.168.47.102]
Oct 31 18:54:26 progressoo postfix/smtpd[22518]: disconnect from anna.infra.opensuse.org[192.168.47.102] quit=1 commands=1

Yeah, I just ran a tcpdump - I think this is my fault. I changed the smtp_bind_address on anna - I'm going to comment out for now, then look at it tomorrow.

Actions #18

Updated by pjessen 6 months ago

pjessen wrote in #note-17:

Yeah, I just ran a tcpdump - I think this is my fault. I changed the smtp_bind_address on anna - I'm going to comment out for now, then look at it tomorrow.

Mea culpa. I was too fast - the bind address really does need setting, but only for external deliveries.

Actions #19

Updated by pjessen 5 months ago

  • Related to tickets #124793: permission denied. Command output: Request was denied by your Redmine server. added
Actions #20

Updated by pjessen 5 months ago

  • Subject changed from progressoo - need proper envelope bounce address for redmine to anna/elsa - which smtp_bind_address to use?

Clearly "tomorrow" got delayed. Just in case, let me summarise the situation.

Postfix on "anna" does currently not use a specific bind address. For deliveries, this means just using the first address, ipv4 or ipv6, depending on the destination host.
Right now, for delivery to an IPv6 address, "anna" would use 2001:67c:2178:8::18 = static.opensuse.org
For delivery to an IPv4 address, "anna" would use 195.135.221.145 = proxy-nue1.opensuse.org

I believe this is the appropriate configuration:

smtp_bind_address  = 195.135.221.145
smtp_bind_address6 = 2001:67c:2178:8::145

However, for internal deliveries, e.g. to progressoo.infra.o.o, this would not work. I think the easy solution is a transport-map:

.opensuse.org   smtpinfra:

and add a transport to master.cf:

smtpinfra   unix  -       -       n       -       10      smtp
    -o smtp_bind_address=

I am specifically not implementing this right now, I see a few unusual things going on, on anna.

Actions #21

Updated by crameleon 3 months ago

  • Status changed from Feedback to In Progress
Actions #22

Updated by crameleon 3 months ago

  • Status changed from In Progress to Resolved

Applied.

Actions

Also available in: Atom PDF