openSUSE admin

cboltz wrote:

I'm afraid there are only two ways how we can fix that:

• stop using SPF, DKIM and DMARC for opensuse.org - or -
• provide a SMTP server the members can use

That Google (or any other provier) rejects Emails from dynamic IP addresses is not really new - and even wanted to avoid SPAM.

As some providers are meanwhile not accepting Emails from Domains without DKIM support (or give them a SPAM rating), I suggest that we might consider the 2nd option as a possible solution. DKIM is also the interim step towards DMARC, which is already requested for openSUSE since a while now.

I think we should start on the heroes ML (or in the meeting next week) and decide which way we want to choose.

If we decide to provide a SMTP server for sending mails as $member@o.o, this is of course something to announce on the project ML as soon as it's ready.

Oh, forgot to mention - no dynamic IP involved. My mail server lives at Hetzner with a static IP, reverse DNS etc.

cboltz wrote:

I tried to send a mail with cboltz@o.o as sender (using my own mail server), but gmail rejected it:

FWIW, I have just now tried the same, to my own gmail address - no problems.

Looking at the reports on admin-auto, it seems I'm not the only one who suffers from this problem.

You mean the dmarc reports?

I'm afraid there are only two ways how we can fix that:

1. stop using SPF, DKIM and DMARC for opensuse.org - or -
2. provide a SMTP server the members can use

I vote yes to (1) and no to (2).

(1) I don't believe we are gaining anything.
(2) in principle easy, in practice it's a lot of effort.

lrupp wrote:

As some providers are meanwhile not accepting Emails from Domains without DKIM support (or give them a SPAM rating),
I suggest that we might consider the 2nd option as a possible solution. DKIM is also the interim step towards DMARC,
which is already requested for openSUSE since a while now.

If whoever is requesting it is also funding the effort, I'll be happy to make an offer for the contract. (no smiley).

cboltz wrote:

We discussed this in the meeting, and agreed that we are not keen on setting up a mail relay for the Members ;-)
This also means we'll need to relax or drop the SPF etc. entries so that our Members can continue to send mails as $username@o.o. Lars, can you please do that?

I would be tempted to set up SPF to explicitly allow from anywhere. Or anywhere with a proper DNS setup (reverse->forward->reverse). I'm not sure if SPF even has such an option :-)

I also just had a mail rejected by google:

<someone@gmail.com>: host gmail-smtp-in.l.google.com[64.233.184.26] said:
    550-5.7.26 This message does not have authentication information or fails
    to 550-5.7.26 pass authentication checks. To best protect our users from
    spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
    https://support.google.com/mail/answer/81126#authentication for more 550
    5.7.26 information. 17-20020a05600c029100b0038ecfc8e76fsi7384276wmk.33 -
    gsmtp (in reply to end of DATA command)

Sent from my opensuse.org address, now resending from jessen.ch.

DMARC¶

I just checked the DMARC DNS entries:

v=DMARC1; p=none; pct=100; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m

^{^} Please note the "Policy=none" section. DMARC was already disabled. Only reports where sent.
I enhanced this a bit further now and set:

v=DMARC1; p=none; pct=0; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m

Which also tells the receiving side that 0 (zero) percent of the Emails should fall under the 'none' category (which is already the lowest one). So whatever brings Google to reject your Emails seems not to be related to DMARC.

SPF¶

Regarding SPF we have:

v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~all

According to Google's documentation, the ~all should result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets. We could switch here to -all to completely disable SPF (like DMARC) as well, but I wonder why these kind of rejects just happen now, while we have the SPF records since (at least) 2021-09-17 (which is our oldest backup). I assume, we have this SPF setting even longer.

DKIM¶

Regarding DKIM: this is IMHO not implemented in any openSUSE domain.

Any thoughts?

lrupp wrote:

SPF¶

Regarding SPF we have:

v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~all

According to Google's documentation, the ~all should result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets. We could switch here to -all to completely disable SPF (like DMARC) as well,

Correct me if I'm wrong, but -all would mean to reject mails from other servers, which would make sending mails as cboltz@o.o from my own mail server impossible.

IMHO ?all or even +all would be a better choice.

but I wonder why these kind of rejects just happen now, while we have the SPF records since (at least) 2021-09-17 (which is our oldest backup). I assume, we have this SPF setting even longer.

The most boring explanation I can offer is: I don't use my @o.o address too often, so it might be that I didn't send a mail with the @o.o sender to gmail for a year or two.

lrupp wrote:

## DMARC

I just checked the DMARC DNS entries:

I would be tempted not to publish any DMARC policy at all, but there might be some best-practices information on that.

## SPF

Regarding SPF we have:

v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~all

According to Google's documentation, the ~all should result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets.

Yes, a softfail is suspicious, but not normally a real problem.

We could switch here to -all to completely disable SPF (like DMARC) as well,

I think the ~all is okay - I see other email-alias providers (ieee.org, acm.org) using the same.
Or as Christian suggested, simply "v=spf1 +all" - although that is the same as not publishing any SPF info.

~ means "softfail" and makes it likely to get the mail tagged as spam (or even rejected, like the mail that triggered this ticket), and that's not what should happen to mails sent by our Members. Therefore I'd really prefer ?all (= "neutral") or +all (= "pass"). Since we expect our Members to use their own mailservers, +all would even be "technically correct" ;-)

BTW: Ubuntu (who also provide mail aliases for their members) have v=spf1 include:canonical.com +all

cboltz wrote:

~ means "softfail" and makes it likely to get the mail tagged as spam (or even rejected, like the mail that triggered this ticket), and that's not what should happen to mails sent by our Members.

+1

Therefore I'd really prefer ?all (= "neutral") or +all (= "pass"). Since we expect our Members to use their own mailservers, +all would even be "technically correct" ;-)

+1

BTW: Ubuntu (who also provide mail aliases for their members) have v=spf1 include:canonical.com +all

Nice example. I suggest we follow it.

FWIW, I had two emails to gmail rejected today, same "explanation".

lrupp wrote:

1) DKIM and DMARC are out of this discussion. We are only talking about the SPF entries here.

Why are they out of this disucssion? What is new here is the DMARC policy, I don't recall any rejects by google before we published a DMARC policy ?
My gut feeling say it is the above that is causing the problems. Granted, it could be google having changed policies too, but as our SPF entry did not change, I don't think the problem is SPF.

3) General
I could think about a specific (additional) SMTP server, which could be used by our openSUSE-members for sending their opensuse.org-Emails by authenticating to this server. But I don't see me here as the one who should make the final decision, so leaving it up to "the crowd" ;-)

It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.

Maybe we could disable our DMARC policy (i.e. remove it) for a month, and run some tests to see if we are still getting bounces?

It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.

I think, the SMTP server would only allow accounts from openSUSE members that send with their assigned @opensuse.org alias. So random spammers that just signed up for a new account would not be allowed to relay their mails.

And without SPF/DMARC, spammers did not even need to bother with an account, but could just send from anywhere and there was no way for receivers to tell that this is not us doing it.

bmwiedemann wrote:

It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.

I think, the SMTP server would only allow accounts from openSUSE members that send with their assigned @opensuse.org alias.

Yes certainly, but that is not the point. Maybe see my posting to the heroes list from the beginning of April:
https://lists.opensuse.org/archives/list/heroes@lists.opensuse.org/thread/4A7ETASHCJB5Z64LSLTTXVFMW7RRP5LD/

Here is the not so easy part:

With hundreds of user credentials spread around the world in uncontrolled/insecure locations, the risk of one getting compromised
or "borrowed" cannot be ignored.  It is a spammer's wet dream - access to a mailserver with DMARC authentication.

If our outgoing mailserver is seen to be spamming, it won't be long before no one can send anything at all.  To mitigate this risk, we will
need rate controls and checks on userids being used from multiple locations. A lot less easy, but still it can be done.

The question is - is it worth it ?

And without SPF/DMARC, spammers did not even need to bother with an account, but could just send from anywhere
and there was no way for receivers to tell that this is not us doing it.

It isn't us :-) That is the current situation and that is what we want, in my opinion.

It is a spammer's wet dream - access to a mailserver with DMARC authentication.

It is pretty common these days for spammers to register their own domain and have valid DKIM+DMARC for it. It is probably part of hosters' standard deployment.
E.g. I got one from @chimeicorpplc.pw

So not really that bad. If it happens, the user gets assigned a new PW.

Alternatively we hand out SSL client certs and if one leaks, it gets added to the CRL.

bmwiedemann wrote:

It is a spammer's wet dream - access to a mailserver with DMARC authentication.

It is pretty common these days for spammers to register their own domain and have valid DKIM+DMARC for it. It is probably part of hosters' standard deployment.
E.g. I got one from @chimeicorpplc.pw

So not really that bad. If it happens, the user gets assigned a new PW.

I guess we will discover it when our mailserver has been blacklisted.

Alternatively we hand out SSL client certs and if one leaks, it gets added to the CRL.

I have yet to hear the business case for all of this effort. It seems like much ado about nothing. Anyway, enough from me, I am opposed to it.

As just discussed and decided in the Heroes meeting, I changed the SPF entry to +all. In theory this should make things better ;-)

I just had two emails rejected by google:

<helpdesk@clarkson.edu>: host aspmx.l.google.com[173.194.76.27] said: 550-5.7.1
    [88.198.198.124      12] Our system has detected that this message is
    550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to
    Gmail, 550-5.7.1 this message has been blocked. Please visit 550-5.7.1
    https://support.google.com/mail/?p=UnsolicitedMessageError 550 5.7.1  for
    more information. o3-20020a05600c2e0300b0039446b4a065si1496380wmf.30 -
    gsmtp (in reply to end of DATA command)

<medwinz@gmail.com>: host gmail-smtp-in.l.google.com[173.194.76.26] said:

    550-5.7.26 This message does not have authentication information or fails
    to 550-5.7.26 pass authentication checks. To best protect our users from
    spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
    https://support.google.com/mail/answer/81126#authentication for more 550
    5.7.26 information. k5-20020a056000004500b00207a5488326si12156215wrx.446 -
    gsmtp (in reply to end of DATA command)

I'll try sending them both from my private address, per@jessen.ch

pjessen wrote:

Looking up one of the URLs suggested in the rejection message, and google suggests signing up for "Postmaster Tools".

In "Postmaster Tools" there is a dashboard with various bits of information, reputation etc. I don't immediately see that anything is wrong. I can add other users, let me know (offline) which addresses to add (I don't know if they have to be google accounts, but probably not).

Another bounce:

<ftpadmin@g.unicamp.br>: host aspmx.l.google.com[66.102.1.27] said: 550-5.7.1
    [88.198.172.124      12] Our system has detected that this message is
    550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to
    Gmail, 550-5.7.1 this message has been blocked. Please visit 550-5.7.1
    https://support.google.com/mail/?p=UnsolicitedMessageError 550 5.7.1  for
    more information. y3-20020a5d6203000000b00207a6a693dcsi1693973wru.277 -
    gsmtp (in reply to end of DATA command)

I'll resend from my private address.

And another one later:

<apoio@ccuec.unicamp.br>: host aspmx.l.google.com[64.233.167.27] said:
    550-5.7.1 [88.198.172.124      12] Our system has detected that this
    message is 550-5.7.1 likely unsolicited mail. To reduce the amount of spam
    sent to Gmail, 550-5.7.1 this message has been blocked. Please visit
    550-5.7.1  https://support.google.com/mail/?p=UnsolicitedMessageError 550
    5.7.1  for more information.
    a6-20020a05600c348600b003947b59dfeesi222965wmq.8 - gsmtp (in reply to end
    of DATA command)

It may be a bit premature, but it seems to me we can conclude that our +all SPF policy doesn't work. When I send from @jessen.ch, there is no problem. (jessen.ch has an SPF record, but no DMARC policy).

I have added my own mailserver mail.hostsuisse.com to the 'opensuse.org' SPF record, just as an experiment.
It has the following two addresses, and only mails from 'per@opensuse.org' will be sent.

185.85.248.6
2a03:7520:4c68::6

pjessen wrote:

I have added my own mailserver mail.hostsuisse.com to the 'opensuse.org' SPF record, just as an experiment.

On test#1, it worked! This is obviously not a solution, but seems to indicate that +all does not quite work.

pjessen wrote:

pjessen wrote:

I have added my own mailserver mail.hostsuisse.com to the 'opensuse.org' SPF record, just as an experiment.

On test#1, it worked! This is obviously not a solution, but seems to indicate that +all does not quite work.

I did another few tests, all worked. I have now reduced the SPF record :

from v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx +all

to v=spf1 all

So far that has worked too.

Aaaaaarg:

<justin.w.xd@gmail.com>: host gmail-smtp-in.l.google.com[173.194.76.27] said:
    550-5.7.26 This message does not have authentication information or fails
    to 550-5.7.26 pass authentication checks. To best protect our users from
    spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
    https://support.google.com/mail/answer/81126#authentication for more 550
    5.7.26 information. g5-20020adfd1e5000000b0020ad1f6b8cfsi1081934wrd.635 -
    gsmtp (in reply to end of DATA command)

I had our DMARC policy checked at https://www.dmarcanalyzer.com :

DNS record: v=DMARC1; p=none; pct=0; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m

Problems:

We've noticed the following problems with your DMARC records .
Error : Invalid "pct" value given in combination with "none" policy ("0" given, only 100 is supported).
Warning : There is a size limit present in your "rua" tag: mailto:admin-auto@opensuse.org!5m. It is advised to avoid using this for now as it is not supported by all ISP's and could lead to missing reports.
Warning : There is a size limit present in your "ruf" tag: mailto:admin-auto@opensuse.org!5m. It is advised to avoid using this for now as it is not supported by all ISP's and could lead to missing reports.
Info : We have found a custom address in your DMARC record. Please make sure to forward the incoming reports to your custom address to see the statistics in the tool.

The report about the "pct" value sounds like it could be a real issue, I'll change it to 100.

I guess this is progress:

<justin.w.xd@gmail.com>: host alt1.gmail-smtp-in.l.google.com[142.250.153.27]
    said: 451-4.7.24 [88.198.198.124      15] The SPF record of the sending
    domain has one 451-4.7.24 or more suspicious entries. To protect our users
    from spam, mail 451-4.7.24 sent from your IP address has been temporarily
    rate limited. Please 451-4.7.24 visit 451-4.7.24
    https://support.google.com/mail/answer/81126#authentication for more 451
    4.7.24 information. qa18-20020a170907869200b006f4405ff418si5420573ejc.453 -
    gsmtp (in reply to end of DATA command)

I can only presume that "one or more suspicious entries" in the SPF record is our "all". The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable". I have changed our SPF record to say ?all.

pjessen wrote:

I can only presume that "one or more suspicious entries" in the SPF record is our "all".
The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable".
I have changed our SPF record to say ?all.

I have to admit that I'm not a big fan of playing around with productive systems all the time: can I suggest that you use one of the alternative DNS domains for your SPAM testing?

We have enough of them:
opensuse-project.com
opensuse-project.de
opensuse-project.net
opensuse-project.org
opensuse.asia
opensuse.co.in
opensuse.co
opensuse.com.br
opensuse.com.es
opensuse.com.mx
opensuse.com
opensuse.de
opensuse.eu
opensuse.fr
opensuse.gen.tr
opensuse.jp
opensuse.kr
opensuse.me
opensuse.mu
opensuse.mx
opensuse.net
opensuse.org.cn
opensuse.org
opensuse.pk

Regards,
Lars

lrupp wrote:

pjessen wrote:

I can only presume that "one or more suspicious entries" in the SPF record is our "all".
The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable".
I have changed our SPF record to say ?all.

I have to admit that I'm not a big fan of playing around with productive systems all the time: can I suggest that you use one of the alternative DNS domains for your SPAM testing?

Yeah, that is probably not a bad idea, although we should perhaps have done with the DMARC setup too :-)

Fwiw, I have now had two mails successfully delivered to Google, one to a hosted customer (clarkson.edu) and one to a plain gmail user.

Looking at anna today, I noticed a lot of gmail bounces, with the usual explanation. I really do not understand why Google will bounce some and not others. Looking at the log since midnight, I see mails to the same address being accepted (19), other times being bounced (6). All with the simplified SPF setting v=spf1 ?all
In total since midnight, Google bounced 2663 mails and accepted 4493.
To be safe, I have re-instated the original SPF record for "opensuse.org", except I have kept the '?all'. I think we should have a look at the address ranges listed, but that's low priority.

Today on anna, since midnight we have delivered 14072 mails to google. However, 2739 were bounced ...

550-5.7.26 This message does not have authentication information or fails to 
550-5.7.26 pass authentication checks. To best protect our users from spam, the 
550-5.7.26 message has been blocked. Please visit 
550-5.7.26  https://support.google.com/mail/answer/81126#authentication for more 
550 5.7.26 information.

I'm giving up, I really don't know what is going on. I would be tempted to remove the DMARC policy, maybe also the SPF record. (at least as an experiment).

This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org

Summary: 15.41% of all messages to google were bounced

Logfile is /var/log/mail-20220519.xz
Number of messages successfully sent =  20514
Number of messages attempted =  24263
Number of messages deferred (quota)  =      1
Number of messages deferred (rate)   =      0
Number of messages deferred (other)  =      0
Number of messages deferred (total)  =      0
Number of messages bounced (quota)       =    241
Number of messages bounced (nosuchuser)  =    216
Number of messages bounced (userdsbled)  =      0
Number of messages bounced (blocked)     =      0
Number of messages bounced (spam)        =      0
Number of messages bounced (reputation)  =      0
Number of messages bounced (authen)      =   3283
Number of messages bounced (other)       =      0
Number of messages bounced (total)       =   3740

This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org

Summary: 14.51% of all messages to google were bounced

Logfile is /var/log/mail-20220520.xz
Number of messages attempted         =  29743
Number of messages successfully sent =  23711
Number of messages deferred (quota)  =    140
Number of messages deferred (rate)   =      2
Number of messages deferred (other)  =      0
Number of messages deferred (total)  =      0
Number of messages bounced (quota)       =     54
Number of messages bounced (nosuchuser)  =    215
Number of messages bounced (userdsbled)  =      0
Number of messages bounced (blocked)     =      0
Number of messages bounced (spam)        =      0
Number of messages bounced (reputation)  =      1
Number of messages bounced (authen)      =   4047
Number of messages bounced (other)       =      0
Number of messages bounced (total)       =   4317

This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org

Summary: 10.01% of all messages to google were bounced

Logfile is /var/log/mail-20220521.xz
Number of messages attempted         =  26838
Number of messages successfully sent =  19356
Number of messages deferred (quota)  =    270
Number of messages deferred (rate)   =      0
Number of messages deferred (other)  =      0
Number of messages deferred (total)  =    270
Number of messages bounced (quota)       =     26
Number of messages bounced (nosuchuser)  =     77
Number of messages bounced (userdsbled)  =      0
Number of messages bounced (blocked)     =      0
Number of messages bounced (spam)        =      0
Number of messages bounced (reputation)  =      0
Number of messages bounced (authen)      =   2585
Number of messages bounced (other)       =      0
Number of messages bounced (total)       =   2688

Now that I have fixed the issue mailman bounce issue, subscribers are being disabled in droves, see #111191 - 53 gmail or googlemail users disabled in a single day.

This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org

Summary: 11.90% of all messages to google were bounced

Logfile is /var/log/mail-20220522.xz
Number of messages attempted         =  20419
Number of messages successfully sent =  11934
Number of messages deferred (quota)  =    310
Number of messages deferred (rate)   =      0
Number of messages deferred (other)  =      0
Number of messages deferred (total)  =    310
Number of messages bounced (quota)       =      4
Number of messages bounced (nosuchuser)  =     12
Number of messages bounced (userdsbled)  =      0
Number of messages bounced (blocked)     =      0
Number of messages bounced (spam)        =      0
Number of messages bounced (reputation)  =      0
Number of messages bounced (authen)      =   2414
Number of messages bounced (other)       =      0
Number of messages bounced (total)       =   2430

Wild guess - over the last 3-4 days, mails to google have only been bounced by servers with IPv6 addresses.
We list our IPv6 ranges as /48 in the SPF record:

ip6:2001:67c:2178::/48 
ip6:2620:113:8044::/48 
ip6:2a01:138:a004::/48 
ip6:2a07:de40:401::/48

I have to wonder if Google might consider such huge ranges (65,536 x 18'446'744'073'709'551'616 addresses) in an SPF record to be "unsafe"?
I am currently running a small experiment (only delivering outbound mails via ipv4), if that works, I am thinking of reducing the IPv6 ranges in the SPF record to:

ip6:2001:67c:2178::/64 (anna etc)
ip6:2a01:138:a004::/64 (for rsync.o.o)
ip6:2a07:de40:401::/64 (for provo mirror, status2.o.o, maybe others).

pjessen wrote:

I am currently running a small experiment (only delivering outbound mails via ipv4), if that works,

The last bounce by Google was at 10:45 CET. I amended the postfix config an anna at 11:23 CET. Since then, all outgoing mails have been delivered to IPv4 addresses, without a single bounce.

This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org

Summary: 4.29% of all messages to google were bounced

Logfile is /var/log/mail-20220523.xz
Number of messages attempted         =  12916
Number of messages successfully sent =   5712
Number of messages deferred (quota)  =    332
Number of messages deferred (rate)   =      0
Number of messages deferred (other)  =      0
Number of messages deferred (total)  =    332
Number of messages bounced (quota)       =      0
Number of messages bounced (nosuchuser)  =      0
Number of messages bounced (userdsbled)  =      0
Number of messages bounced (blocked)     =      0
Number of messages bounced (spam)        =      0
Number of messages bounced (reputation)  =      0
Number of messages bounced (authen)      =    555
Number of messages bounced (other)       =      0
Number of messages bounced (total)       =    555

pjessen wrote:

pjessen wrote:

I am currently running a small experiment (only delivering outbound mails via ipv4), if that works,

The last bounce by Google was at 10:45 CET. I amended the postfix config on anna at 11:23 CET. Since then, all outgoing mails have been delivered to IPv4 addresses, without a single bounce.

It has only been a bit more than 22 hours, but still no bounces since 10:45 CET yesterday. The 4.29% bounced is also a significant improvement. I think that is a very clear indication that there is an issue with IPv6.

Following Lars' suggestion "don't play with production", I have set up a test scheme using opensuse.mx. It has a separate SPF record:

ip6:2001:67c:2178:8::/64 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48
                  ^^^^^^

I force all outbound mail from opensuse.mx to be sent via ipv6. So far I have sent myself about 100 test mails, no bounces.

pjessen wrote:

• 2620:113:8044::/48 - is this actually being used to send mails from opensuse.org?

It is not mentioned here: https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Network

Oh well. There goes that theory.
Between 1830 and 1839, 131 mails to gmail addresses bounced. I have reverted to delivery by IPv4 only.

When it worked so perfectly fine for opensuse.mx, I have to wonder if it is DMARC related, opensuse.mx does not have a DMARC record. I have added a DMARC record for opensuse.mx.

pjessen wrote:

When it worked so perfectly fine for opensuse.mx, I have to wonder if it is DMARC related, opensuse.mx does not have a DMARC record. I have added a DMARC record for opensuse.mx.

I had hoped mails from opensuse.mx would also be bounced, but they were not. Another wild theory - the "reputation" of opensuse.mx is fine, whereas the "reputation" of opensuse.org is very poor ?

Well, I am leaving anna to send outbound mails over ipv4 only. This is very crude measure, and it affects all recipient addresses. I have also amended the postfix config on mx{12}.o.o to do the same, deliveries only over ipv4. I cannot explain why ipv4 vs ipv6 should make any difference.

This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org

Summary: 0.78% of all messages to google were bounced

Logfile is /var/log/mail-20220524.xz
Number of messages attempted         =  21757
Number of messages successfully sent =  16107
Number of messages deferred (quota)  =   1087
Number of messages deferred (rate)   =    196
Number of messages deferred (other)  = 18446744073709551426
Number of messages deferred (total)  =   1093
Number of messages bounced (quota)       =     11
Number of messages bounced (nosuchuser)  =      3
Number of messages bounced (userdsbled)  =      1
Number of messages bounced (blocked)     =      1
Number of messages bounced (spam)        =      0
Number of messages bounced (reputation)  =      0
Number of messages bounced (authen)      =    154
Number of messages bounced (other)       =      0
Number of messages bounced (total)       =    170

I may have just found out why ipv4 vs ipv6 seems to making a difference here. Having limited delivery to ipv4, we just don't get any bounces from Google. Probably because:

# dig lists.opensuse.org txt
;; ANSWER SECTION:
lists.opensuse.org.     1800    IN      TXT     "v=spf1 mx a:proxy-nue1.opensuse.org a:proxy-nue2.opensuse.org ?all"

# host proxy-nue1.opensuse.org
proxy-nue1.opensuse.org has address 195.135.221.145
# host proxy-nue2.opensuse.org
proxy-nue2.opensuse.org has address 195.135.221.139

So, we basically do not permit outbound list mails over IPv6 ...... aaaaaarg. I can't believe I did not see that earlier. It probably has no impact on $SUBJ though.

Earlier, I amended the SPF record for lists.o.o, adding ip6:2001:67c:2178:8::/64 and making it -all. Mailman is the only one sending anything with sender lists.o.o, and mailman forwards over relay.infra.opensuse.org only. For about three hours, everything to gmail and googlemail has been sent over ipv6, nothing has bounced due to "authentication". Yippee!

To return to $SUBJ, I have been testing with opensuse.mx. At first, I configured the SPF record to have '-all'. Sending a mail from per@opensuse.mx from my own system was correctly bounced by Google:

550-5.7.26 This message fails to pass SPF checks for an SPF record with a hard 
550-5.7.26 failpolicy. To best protect our users from spam and phishing, the 
550-5.7.26 message has been blocked. Please visit 
550-5.7.26  https://support.google.com/mail/answer/81126#authentication for more 
550 5.7.26 information. 

"spf record with hard failpolicy" - sounds absolutely correct.

Then I changed the SPF record to have '?all' and sent about 100 test-mails from per@opensuse.mx to my gmail address. They are still arriving, so far none have bounced.
Currently, opensuse.mx has no DMARC policy.

pjessen wrote:

Earlier, I amended the SPF record for lists.o.o, adding ip6:2001:67c:2178:8::/64 and making it -all. Mailman is the only one sending anything with sender lists.o.o, and mailman forwards over relay.infra.opensuse.org only. For about three hours, everything to gmail and googlemail has been sent over ipv6, nothing has bounced due to "authentication". Yippee!

Nice catch!

However, I don't really like -all because it breaks forwarding. Maybe you could use ~all (softfail) instead to give people who forward their mailinglist mails to another mailbox a little chance?

This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org

Summary: 0.02% of all messages to google were bounced

Logfile is /var/log/mail-20220525.xz
Number of messages attempted         =  42022
Number of messages successfully sent =  29770
Number of messages deferred (quota)  =    718
Number of messages deferred (rate)   =    270
Number of messages deferred (other)  = 18446744073709551368
Number of messages deferred (total)  =    740
Number of messages bounced (quota)       =     10
Number of messages bounced (nosuchuser)  =      0
Number of messages bounced (userdsbled)  =      2
Number of messages bounced (blocked)     =      0
Number of messages bounced (spam)        =      0
Number of messages bounced (reputation)  =      0
Number of messages bounced (authen)      =      0
Number of messages bounced (other)       =      0
Number of messages bounced (total)       =     12

cboltz wrote:

However, I don't really like -all because it breaks forwarding. Maybe you could use ~all (softfail) instead to give people who forward their mailinglist mails to another mailbox a little chance?

In principle, any forwarder ought to use SRS, but yeah why not, I suppose it won't hurt us :-)

pjessen wrote:

To return to $SUBJ, I have been testing with opensuse.mx.

I changed the SPF record to have '?all' and sent about 100 test-mails from per@opensuse.mx to my gmail address. They are still arriving, so far none have bounced.
Currently, opensuse.mx has no DMARC policy.

Comparing opensuse.mx and opensuse.org setup:

o.mx  spf:  v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 ip6:2a01:138:a004::/64 ip6:2a07:de40:401::/64 mx ?all
o.org spf:  v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 ip6:2a01:138:a004::/64 ip6:2a07:de40:401::/64 mx ?all

o.mx dmarc:  none
o.org dmarc: v=DMARC1; p=none; pct=100; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m

o.mx usage:  nothing much at all.
o.org usage: everything.

For testing, I have to remove the DMARC record for opensuse.org, for maybe 24 hours. I hope that is okay. We have a holiday coming up.

pjessen wrote:

For testing, I have to remove the DMARC record for opensuse.org, for maybe 24 hours. I hope that is okay. We have a holiday coming up.

Well, that made no difference. Right now, with the exact same setup, the same mail sent from opensuse.mx and opensuse.org (from my home setup) is accepted and rejected, respectively.
I have re-instated the DMARC record for opensuse.org.

pjessen wrote:

cboltz wrote:

BTW: Ubuntu (who also provide mail aliases for their members) have v=spf1 include:canonical.com +all

Nice example. I suggest we follow it.

I thought I would check again - the SPF record for ubuntu.com remains the same (see above), and interestingly they don't publish any DMARC nor DKIM records. It seems to me they ought to have the same issues we are having.

For my testing with opensuse.mx, I have updated the SPF record to say "+all". I may take a closer look when I have some time.

pjessen wrote:

For my testing with opensuse.mx, I have updated the SPF record to say "+all". I may take a closer look when I have some time.

opensuse.mx has no dmarc nor dkim record.
spf record is v=spf1 include:_spf.opensuse.org +all
_spf.opensuse.org is v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 ip6:2a01:138:a004::/64 ip6:2a07:de40:401::/64 mx ?all

With that config, test mails from per@opensuse.mx to my own gmail address bounced, "suspected spam".

New try:
_spf.opensuse.mx = v=spf1 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 mx -all
This is quite strict, and will intentionally fail anything not coming from an openSUSE IP range. However,
with the SPF record = v=spf1 include:_spf.opensuse.mx +all, I am hoping the '+all' will override the previous -all

pjessen wrote:

New try:
_spf.opensuse.mx = v=spf1 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 mx -all
This is quite strict, and will intentionally fail anything not coming from an openSUSE IP range. However,
with the SPF record = v=spf1 include:_spf.opensuse.mx +all, I am hoping the '+all' will override the previous -all

With the config above, sending a mail from per@opensuse.mx to my gmail address still results in the following bounce:

<redacted@gmail.com>: host gmail-smtp-in.l.google.com[74.125.140.27] said:
550-5.7.1 [88.198.198.124      12] Our system has detected that this message is 
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail, 
550-5.7.1 this message has been blocked. Please visit
550-5.7.1  https://support.google.com/mail/?p=UnsolicitedMessageError 
550 5.7.1  for more information.

I would like if someone would review my setup for opensuse.mx, just to make sure I am not missing something important.

The objective is to enable anyone to send emails with their opensuse alias address, from whereever they are. The +all fall-back SPF policy should enable this, but Google does not accept it.

Currently, Google rejects mails from opensuse.org and opensuse.mx, apparently ignoring the +all fall-back SPF policy.

pjessen wrote:

Currently, Google rejects mails from opensuse.org and opensuse.mx, apparently ignoring the +all fall-back SPF policy.

looks like this is stil the case - and I guess google gives a shit about '+all'?

DocB wrote:

pjessen wrote:

Currently, Google rejects mails from opensuse.org and opensuse.mx, apparently ignoring the +all fall-back SPF policy.

looks like this is stil the case - and I guess google gives a shit about '+all'?

I would be a lot less diplomatic about it ... yes, I wonder if Google actually treats "+all" as even worse.
To be honest, it is so frustrating, I have essentially given up on it.