tickets #109025
opengmail rejects mails from cboltz@o.o - SPF/DKIM/DMARC issue
Added by cboltz over 2 years ago. Updated over 1 year ago.
50%
Description
I tried to send a mail with cboltz@o.o as sender (using my own mail server), but gmail rejected it:
<REDACTED@gmail.com>: host gmail-smtp-in.l.google.com[142.250.153.26] said:
550-5.7.26 This message does not have authentication information or fails
to 550-5.7.26 pass authentication checks. To best protect our users from
spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
https://support.google.com/mail/answer/81126#authentication for more 550
5.7.26 information.
Sending a mail with nearly the same content and using the same server, but using a mail address @cboltz.de as sender worked without problems.
This looks like a problem with the SPF and/or DKIM and/or DMARC settings for opensuse.org. A test mail to https://www.kbxscore.com with cboltz@o.o as sender gets reported as
DKIM: FAIL No DKIM authentication signature found in this email message.
SPF: SOFTFAIL (domain owner discourages use of this host) identity=mailfrom; client-ip=88.99.101.17; helo=mail.cboltz.de; envelope-from=REDACTED@opensuse.org; receiver=REDACTED@kbxscore.com
DMARC: FAIL Uh-oh! Message did not pass DMARC checks. No DKIM signature found for opensuse.org. DMARC policy for opensuse.org: p=none; pct=100; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m
The missing DKIM signature and the SPF softfail are not surprising - since we don't offer a SMTP server for our members to send out mails as $member@o.o, I have to use my own server (which of course can't create a DKIM signature for o.o, and also is outside of the IP range in the SPF entry).
Looking at the reports on admin-auto, it seems I'm not the only one who suffers from this problem.
I'm afraid there are only two ways how we can fix that:
- stop using SPF, DKIM and DMARC for opensuse.org - or -
- provide a SMTP server the members can use
Updated by lrupp over 2 years ago
cboltz wrote:
I'm afraid there are only two ways how we can fix that:
- stop using SPF, DKIM and DMARC for opensuse.org - or -
- provide a SMTP server the members can use
That Google (or any other provier) rejects Emails from dynamic IP addresses is not really new - and even wanted to avoid SPAM.
As some providers are meanwhile not accepting Emails from Domains without DKIM support (or give them a SPAM rating), I suggest that we might consider the 2nd option as a possible solution. DKIM is also the interim step towards DMARC, which is already requested for openSUSE since a while now.
Updated by lrupp over 2 years ago
- Status changed from New to Feedback
- Assignee changed from lrupp to cboltz
IMHO this ticket should be discussed a bit wider. I'm just not sure if it's enough to start a discussion on the heroes mailing list, on the project mailing list or at code.opensuse.org.
=> Christian: what do you think?
Updated by cboltz over 2 years ago
I think we should start on the heroes ML (or in the meeting next week) and decide which way we want to choose.
If we decide to provide a SMTP server for sending mails as $member@o.o, this is of course something to announce on the project ML as soon as it's ready.
Updated by cboltz over 2 years ago
Oh, forgot to mention - no dynamic IP involved. My mail server lives at Hetzner with a static IP, reverse DNS etc.
Updated by pjessen over 2 years ago
cboltz wrote:
I tried to send a mail with cboltz@o.o as sender (using my own mail server), but gmail rejected it:
FWIW, I have just now tried the same, to my own gmail address - no problems.
Looking at the reports on admin-auto, it seems I'm not the only one who suffers from this problem.
You mean the dmarc reports?
I'm afraid there are only two ways how we can fix that:
- stop using SPF, DKIM and DMARC for opensuse.org - or -
- provide a SMTP server the members can use
I vote yes to (1) and no to (2).
(1) I don't believe we are gaining anything.
(2) in principle easy, in practice it's a lot of effort.
Updated by pjessen over 2 years ago
lrupp wrote:
As some providers are meanwhile not accepting Emails from Domains without DKIM support (or give them a SPAM rating),
I suggest that we might consider the 2nd option as a possible solution. DKIM is also the interim step towards DMARC,
which is already requested for openSUSE since a while now.
If whoever is requesting it is also funding the effort, I'll be happy to make an offer for the contract. (no smiley).
Updated by cboltz over 2 years ago
- Status changed from Feedback to New
- Assignee changed from cboltz to lrupp
We discussed this in the meeting, and agreed that we are not keen on setting up a mail relay for the Members ;-)
This also means we'll need to relax or drop the SPF etc. entries so that our Members can continue to send mails as $username@o.o. Lars, can you please do that?
Updated by pjessen over 2 years ago
cboltz wrote:
We discussed this in the meeting, and agreed that we are not keen on setting up a mail relay for the Members ;-)
This also means we'll need to relax or drop the SPF etc. entries so that our Members can continue to send mails as $username@o.o. Lars, can you please do that?
I would be tempted to set up SPF to explicitly allow from anywhere. Or anywhere with a proper DNS setup (reverse->forward->reverse). I'm not sure if SPF even has such an option :-)
Updated by pjessen over 2 years ago
I also just had a mail rejected by google:
<someone@gmail.com>: host gmail-smtp-in.l.google.com[64.233.184.26] said:
550-5.7.26 This message does not have authentication information or fails
to 550-5.7.26 pass authentication checks. To best protect our users from
spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
https://support.google.com/mail/answer/81126#authentication for more 550
5.7.26 information. 17-20020a05600c029100b0038ecfc8e76fsi7384276wmk.33 -
gsmtp (in reply to end of DATA command)
Sent from my opensuse.org address, now resending from jessen.ch.
Updated by lrupp over 2 years ago
DMARC¶
I just checked the DMARC DNS entries:
v=DMARC1; p=none; pct=100; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m
^ Please note the "Policy=none" section. DMARC was already disabled. Only reports where sent.
I enhanced this a bit further now and set:
v=DMARC1; p=none; pct=0; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m
Which also tells the receiving side that 0 (zero) percent of the Emails should fall under the 'none' category (which is already the lowest one). So whatever brings Google to reject your Emails seems not to be related to DMARC.
SPF¶
Regarding SPF we have:
v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~all
According to Google's documentation, the ~all should result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets. We could switch here to -all to completely disable SPF (like DMARC) as well, but I wonder why these kind of rejects just happen now, while we have the SPF records since (at least) 2021-09-17 (which is our oldest backup). I assume, we have this SPF setting even longer.
DKIM¶
Regarding DKIM: this is IMHO not implemented in any openSUSE domain.
Any thoughts?
Updated by cboltz over 2 years ago
lrupp wrote:
SPF¶
Regarding SPF we have:
v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~allAccording to Google's documentation, the
~allshould result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets. We could switch here to-allto completely disable SPF (like DMARC) as well,
Correct me if I'm wrong, but -all would mean to reject mails from other servers, which would make sending mails as cboltz@o.o from my own mail server impossible.
IMHO ?all or even +all would be a better choice.
but I wonder why these kind of rejects just happen now, while we have the SPF records since (at least) 2021-09-17 (which is our oldest backup). I assume, we have this SPF setting even longer.
The most boring explanation I can offer is: I don't use my @o.o address too often, so it might be that I didn't send a mail with the @o.o sender to gmail for a year or two.
Updated by pjessen over 2 years ago
lrupp wrote:
## DMARC
I just checked the DMARC DNS entries:
I would be tempted not to publish any DMARC policy at all, but there might be some best-practices information on that.
## SPF
Regarding SPF we have:
v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~allAccording to Google's documentation, the
~allshould result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets.
Yes, a softfail is suspicious, but not normally a real problem.
We could switch here to
-allto completely disable SPF (like DMARC) as well,
I think the ~all is okay - I see other email-alias providers (ieee.org, acm.org) using the same.
Or as Christian suggested, simply "v=spf1 +all" - although that is the same as not publishing any SPF info.
Updated by cboltz over 2 years ago
~ means "softfail" and makes it likely to get the mail tagged as spam (or even rejected, like the mail that triggered this ticket), and that's not what should happen to mails sent by our Members. Therefore I'd really prefer ?all (= "neutral") or +all (= "pass"). Since we expect our Members to use their own mailservers, +all would even be "technically correct" ;-)
BTW: Ubuntu (who also provide mail aliases for their members) have v=spf1 include:canonical.com +all
Updated by pjessen over 2 years ago
cboltz wrote:
~means "softfail" and makes it likely to get the mail tagged as spam (or even rejected, like the mail that triggered this ticket), and that's not what should happen to mails sent by our Members.
+1
Therefore I'd really prefer
?all(= "neutral") or+all(= "pass"). Since we expect our Members to use their own mailservers,+allwould even be "technically correct" ;-)
+1
BTW: Ubuntu (who also provide mail aliases for their members) have
v=spf1 include:canonical.com +all
Nice example. I suggest we follow it.
Updated by pjessen over 2 years ago
FWIW, I had two emails to gmail rejected today, same "explanation".
Updated by lrupp over 2 years ago
- Status changed from New to Workable
- Assignee changed from lrupp to opensuse-admin
1) DKIM and DMARC are out of this discussion. We are only talking about the SPF entries here.
2) SPF
In the past, opensuse.org often enough ended up on one of the public blacklists, as we allowed everyone (including Spammers) to submit his @opensuse.org EMail from everywhere. Since we added the ~all SPF record, opensuse.org was not ending on one single blacklist any more.
Therefor, I vote for keeping the entry as it is. But that's just me (see below).
3) General
I could think about a specific (additional) SMTP server, which could be used by our openSUSE-members for sending their opensuse.org-Emails by authenticating to this server. But I don't see me here as the one who should make the final decision, so leaving it up to "the crowd" ;-)
Updated by pjessen over 2 years ago
lrupp wrote:
1) DKIM and DMARC are out of this discussion. We are only talking about the SPF entries here.
Why are they out of this disucssion? What is new here is the DMARC policy, I don't recall any rejects by google before we published a DMARC policy ?
My gut feeling say it is the above that is causing the problems. Granted, it could be google having changed policies too, but as our SPF entry did not change, I don't think the problem is SPF.
3) General
I could think about a specific (additional) SMTP server, which could be used by our openSUSE-members for sending their opensuse.org-Emails by authenticating to this server. But I don't see me here as the one who should make the final decision, so leaving it up to "the crowd" ;-)
It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.
Maybe we could disable our DMARC policy (i.e. remove it) for a month, and run some tests to see if we are still getting bounces?
Updated by bmwiedemann over 2 years ago
It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.
I think, the SMTP server would only allow accounts from openSUSE members that send with their assigned @opensuse.org alias. So random spammers that just signed up for a new account would not be allowed to relay their mails.
And without SPF/DMARC, spammers did not even need to bother with an account, but could just send from anywhere and there was no way for receivers to tell that this is not us doing it.
Updated by pjessen over 2 years ago
bmwiedemann wrote:
It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.
I think, the SMTP server would only allow accounts from openSUSE members that send with their assigned @opensuse.org alias.
Yes certainly, but that is not the point. Maybe see my posting to the heroes list from the beginning of April:
https://lists.opensuse.org/archives/list/heroes@lists.opensuse.org/thread/4A7ETASHCJB5Z64LSLTTXVFMW7RRP5LD/
Here is the not so easy part:
With hundreds of user credentials spread around the world in uncontrolled/insecure locations, the risk of one getting compromised
or "borrowed" cannot be ignored. It is a spammer's wet dream - access to a mailserver with DMARC authentication.
If our outgoing mailserver is seen to be spamming, it won't be long before no one can send anything at all. To mitigate this risk, we will
need rate controls and checks on userids being used from multiple locations. A lot less easy, but still it can be done.
The question is - is it worth it ?
And without SPF/DMARC, spammers did not even need to bother with an account, but could just send from anywhere
and there was no way for receivers to tell that this is not us doing it.
It isn't us :-) That is the current situation and that is what we want, in my opinion.
Updated by bmwiedemann over 2 years ago
It is a spammer's wet dream - access to a mailserver with DMARC authentication.
It is pretty common these days for spammers to register their own domain and have valid DKIM+DMARC for it. It is probably part of hosters' standard deployment.
E.g. I got one from @chimeicorpplc.pw
So not really that bad. If it happens, the user gets assigned a new PW.
Alternatively we hand out SSL client certs and if one leaks, it gets added to the CRL.
Updated by pjessen over 2 years ago
bmwiedemann wrote:
It is a spammer's wet dream - access to a mailserver with DMARC authentication.
It is pretty common these days for spammers to register their own domain and have valid DKIM+DMARC for it. It is probably part of hosters' standard deployment.
E.g. I got one from @chimeicorpplc.pwSo not really that bad. If it happens, the user gets assigned a new PW.
I guess we will discover it when our mailserver has been blacklisted.
Alternatively we hand out SSL client certs and if one leaks, it gets added to the CRL.
I have yet to hear the business case for all of this effort. It seems like much ado about nothing. Anyway, enough from me, I am opposed to it.
Updated by cboltz over 2 years ago
As just discussed and decided in the Heroes meeting, I changed the SPF entry to +all. In theory this should make things better ;-)
Updated by pjessen over 2 years ago
I just had two emails rejected by google:
<helpdesk@clarkson.edu>: host aspmx.l.google.com[173.194.76.27] said: 550-5.7.1
[88.198.198.124 12] Our system has detected that this message is
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to
Gmail, 550-5.7.1 this message has been blocked. Please visit 550-5.7.1
https://support.google.com/mail/?p=UnsolicitedMessageError 550 5.7.1 for
more information. o3-20020a05600c2e0300b0039446b4a065si1496380wmf.30 -
gsmtp (in reply to end of DATA command)
<medwinz@gmail.com>: host gmail-smtp-in.l.google.com[173.194.76.26] said:
550-5.7.26 This message does not have authentication information or fails
to 550-5.7.26 pass authentication checks. To best protect our users from
spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
https://support.google.com/mail/answer/81126#authentication for more 550
5.7.26 information. k5-20020a056000004500b00207a5488326si12156215wrx.446 -
gsmtp (in reply to end of DATA command)
I'll try sending them both from my private address, per@jessen.ch
Updated by pjessen over 2 years ago
- Status changed from Workable to Feedback
pjessen wrote:
I'll try sending them both from my private address, per@jessen.ch
They both went through. Looking up one of the URLs suggested in the rejection message, and google suggests signing up for "Postmaster Tools".
This will require amending the DNS setup, adding a verification record - does anyone see a problem if I go ahead and do that, using my personal Google account?
Updated by pjessen over 2 years ago
- Status changed from Feedback to In Progress
pjessen wrote:
Looking up one of the URLs suggested in the rejection message, and google suggests signing up for "Postmaster Tools".
This will require amending the DNS setup, adding a verification record - does anyone see a problem if I go ahead and do that, using my personal Google account?
I have added the google verification TXT record.
Updated by pjessen over 2 years ago
pjessen wrote:
Looking up one of the URLs suggested in the rejection message, and google suggests signing up for "Postmaster Tools".
In "Postmaster Tools" there is a dashboard with various bits of information, reputation etc. I don't immediately see that anything is wrong. I can add other users, let me know (offline) which addresses to add (I don't know if they have to be google accounts, but probably not).
Updated by pjessen over 2 years ago
Another bounce:
<ftpadmin@g.unicamp.br>: host aspmx.l.google.com[66.102.1.27] said: 550-5.7.1
[88.198.172.124 12] Our system has detected that this message is
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to
Gmail, 550-5.7.1 this message has been blocked. Please visit 550-5.7.1
https://support.google.com/mail/?p=UnsolicitedMessageError 550 5.7.1 for
more information. y3-20020a5d6203000000b00207a6a693dcsi1693973wru.277 -
gsmtp (in reply to end of DATA command)
I'll resend from my private address.
And another one later:
<apoio@ccuec.unicamp.br>: host aspmx.l.google.com[64.233.167.27] said:
550-5.7.1 [88.198.172.124 12] Our system has detected that this
message is 550-5.7.1 likely unsolicited mail. To reduce the amount of spam
sent to Gmail, 550-5.7.1 this message has been blocked. Please visit
550-5.7.1 https://support.google.com/mail/?p=UnsolicitedMessageError 550
5.7.1 for more information.
a6-20020a05600c348600b003947b59dfeesi222965wmq.8 - gsmtp (in reply to end
of DATA command)
Updated by pjessen over 2 years ago
It may be a bit premature, but it seems to me we can conclude that our +all SPF policy doesn't work. When I send from @jessen.ch, there is no problem. (jessen.ch has an SPF record, but no DMARC policy).
I have added my own mailserver mail.hostsuisse.com to the 'opensuse.org' SPF record, just as an experiment.
It has the following two addresses, and only mails from 'per@opensuse.org' will be sent.
185.85.248.6
2a03:7520:4c68::6
Updated by pjessen over 2 years ago
pjessen wrote:
I have added my own mailserver
mail.hostsuisse.comto the 'opensuse.org' SPF record, just as an experiment.
On test#1, it worked! This is obviously not a solution, but seems to indicate that +all does not quite work.
Updated by pjessen over 2 years ago
pjessen wrote:
pjessen wrote:
I have added my own mailserver
mail.hostsuisse.comto the 'opensuse.org' SPF record, just as an experiment.On test#1, it worked! This is obviously not a solution, but seems to indicate that
+alldoes not quite work.
I did another few tests, all worked. I have now reduced the SPF record :
from v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx +all
to v=spf1 all
So far that has worked too.
Updated by pjessen over 2 years ago
Aaaaaarg:
<justin.w.xd@gmail.com>: host gmail-smtp-in.l.google.com[173.194.76.27] said:
550-5.7.26 This message does not have authentication information or fails
to 550-5.7.26 pass authentication checks. To best protect our users from
spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
https://support.google.com/mail/answer/81126#authentication for more 550
5.7.26 information. g5-20020adfd1e5000000b0020ad1f6b8cfsi1081934wrd.635 -
gsmtp (in reply to end of DATA command)
Updated by pjessen over 2 years ago
I had our DMARC policy checked at https://www.dmarcanalyzer.com :
DNS record: v=DMARC1; p=none; pct=0; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m
Problems:
We've noticed the following problems with your DMARC records .
Error : Invalid "pct" value given in combination with "none" policy ("0" given, only 100 is supported).
Warning : There is a size limit present in your "rua" tag: mailto:admin-auto@opensuse.org!5m. It is advised to avoid using this for now as it is not supported by all ISP's and could lead to missing reports.
Warning : There is a size limit present in your "ruf" tag: mailto:admin-auto@opensuse.org!5m. It is advised to avoid using this for now as it is not supported by all ISP's and could lead to missing reports.
Info : We have found a custom address in your DMARC record. Please make sure to forward the incoming reports to your custom address to see the statistics in the tool.
The report about the "pct" value sounds like it could be a real issue, I'll change it to 100.
Updated by pjessen over 2 years ago
I guess this is progress:
<justin.w.xd@gmail.com>: host alt1.gmail-smtp-in.l.google.com[142.250.153.27]
said: 451-4.7.24 [88.198.198.124 15] The SPF record of the sending
domain has one 451-4.7.24 or more suspicious entries. To protect our users
from spam, mail 451-4.7.24 sent from your IP address has been temporarily
rate limited. Please 451-4.7.24 visit 451-4.7.24
https://support.google.com/mail/answer/81126#authentication for more 451
4.7.24 information. qa18-20020a170907869200b006f4405ff418si5420573ejc.453 -
gsmtp (in reply to end of DATA command)
I can only presume that "one or more suspicious entries" in the SPF record is our "all". The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable". I have changed our SPF record to say ?all.
Updated by lrupp over 2 years ago
pjessen wrote:
I can only presume that "one or more suspicious entries" in the SPF record is our "all".
The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable".
I have changed our SPF record to say?all.
I have to admit that I'm not a big fan of playing around with productive systems all the time: can I suggest that you use one of the alternative DNS domains for your SPAM testing?
We have enough of them:
opensuse-project.com
opensuse-project.de
opensuse-project.net
opensuse-project.org
opensuse.asia
opensuse.co.in
opensuse.co
opensuse.com.br
opensuse.com.es
opensuse.com.mx
opensuse.com
opensuse.de
opensuse.eu
opensuse.fr
opensuse.gen.tr
opensuse.jp
opensuse.kr
opensuse.me
opensuse.mu
opensuse.mx
opensuse.net
opensuse.org.cn
opensuse.org
opensuse.pk
Regards,
Lars
Updated by pjessen over 2 years ago
lrupp wrote:
pjessen wrote:
I can only presume that "one or more suspicious entries" in the SPF record is our "all".
The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable".
I have changed our SPF record to say?all.I have to admit that I'm not a big fan of playing around with productive systems all the time: can I suggest that you use one of the alternative DNS domains for your SPAM testing?
Yeah, that is probably not a bad idea, although we should perhaps have done with the DMARC setup too :-)
Fwiw, I have now had two mails successfully delivered to Google, one to a hosted customer (clarkson.edu) and one to a plain gmail user.
Updated by pjessen over 2 years ago
Looking at anna today, I noticed a lot of gmail bounces, with the usual explanation. I really do not understand why Google will bounce some and not others. Looking at the log since midnight, I see mails to the same address being accepted (19), other times being bounced (6). All with the simplified SPF setting v=spf1 ?all
In total since midnight, Google bounced 2663 mails and accepted 4493.
To be safe, I have re-instated the original SPF record for "opensuse.org", except I have kept the '?all'. I think we should have a look at the address ranges listed, but that's low priority.
Updated by pjessen over 2 years ago
Today on anna, since midnight we have delivered 14072 mails to google. However, 2739 were bounced ...
550-5.7.26 This message does not have authentication information or fails to
550-5.7.26 pass authentication checks. To best protect our users from spam, the
550-5.7.26 message has been blocked. Please visit
550-5.7.26 https://support.google.com/mail/answer/81126#authentication for more
550 5.7.26 information.
I'm giving up, I really don't know what is going on. I would be tempted to remove the DMARC policy, maybe also the SPF record. (at least as an experiment).
Updated by pjessen over 2 years ago
This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org
Summary: 15.41% of all messages to google were bounced
Logfile is /var/log/mail-20220519.xz
Number of messages successfully sent = 20514
Number of messages attempted = 24263
Number of messages deferred (quota) = 1
Number of messages deferred (rate) = 0
Number of messages deferred (other) = 0
Number of messages deferred (total) = 0
Number of messages bounced (quota) = 241
Number of messages bounced (nosuchuser) = 216
Number of messages bounced (userdsbled) = 0
Number of messages bounced (blocked) = 0
Number of messages bounced (spam) = 0
Number of messages bounced (reputation) = 0
Number of messages bounced (authen) = 3283
Number of messages bounced (other) = 0
Number of messages bounced (total) = 3740
Updated by pjessen over 2 years ago
This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org
Summary: 14.51% of all messages to google were bounced
Logfile is /var/log/mail-20220520.xz
Number of messages attempted = 29743
Number of messages successfully sent = 23711
Number of messages deferred (quota) = 140
Number of messages deferred (rate) = 2
Number of messages deferred (other) = 0
Number of messages deferred (total) = 0
Number of messages bounced (quota) = 54
Number of messages bounced (nosuchuser) = 215
Number of messages bounced (userdsbled) = 0
Number of messages bounced (blocked) = 0
Number of messages bounced (spam) = 0
Number of messages bounced (reputation) = 1
Number of messages bounced (authen) = 4047
Number of messages bounced (other) = 0
Number of messages bounced (total) = 4317
Updated by pjessen over 2 years ago
This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org
Summary: 10.01% of all messages to google were bounced
Logfile is /var/log/mail-20220521.xz
Number of messages attempted = 26838
Number of messages successfully sent = 19356
Number of messages deferred (quota) = 270
Number of messages deferred (rate) = 0
Number of messages deferred (other) = 0
Number of messages deferred (total) = 270
Number of messages bounced (quota) = 26
Number of messages bounced (nosuchuser) = 77
Number of messages bounced (userdsbled) = 0
Number of messages bounced (blocked) = 0
Number of messages bounced (spam) = 0
Number of messages bounced (reputation) = 0
Number of messages bounced (authen) = 2585
Number of messages bounced (other) = 0
Number of messages bounced (total) = 2688
Updated by pjessen over 2 years ago
Now that I have fixed the issue mailman bounce issue, subscribers are being disabled in droves, see #111191 - 53 gmail or googlemail users disabled in a single day.
Updated by pjessen over 2 years ago
This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org
Summary: 11.90% of all messages to google were bounced
Logfile is /var/log/mail-20220522.xz
Number of messages attempted = 20419
Number of messages successfully sent = 11934
Number of messages deferred (quota) = 310
Number of messages deferred (rate) = 0
Number of messages deferred (other) = 0
Number of messages deferred (total) = 310
Number of messages bounced (quota) = 4
Number of messages bounced (nosuchuser) = 12
Number of messages bounced (userdsbled) = 0
Number of messages bounced (blocked) = 0
Number of messages bounced (spam) = 0
Number of messages bounced (reputation) = 0
Number of messages bounced (authen) = 2414
Number of messages bounced (other) = 0
Number of messages bounced (total) = 2430
Updated by pjessen over 2 years ago
Wild guess - over the last 3-4 days, mails to google have only been bounced by servers with IPv6 addresses.
We list our IPv6 ranges as /48 in the SPF record:
ip6:2001:67c:2178::/48
ip6:2620:113:8044::/48
ip6:2a01:138:a004::/48
ip6:2a07:de40:401::/48
I have to wonder if Google might consider such huge ranges (65,536 x 18'446'744'073'709'551'616 addresses) in an SPF record to be "unsafe"?
I am currently running a small experiment (only delivering outbound mails via ipv4), if that works, I am thinking of reducing the IPv6 ranges in the SPF record to:
ip6:2001:67c:2178::/64 (anna etc)
ip6:2a01:138:a004::/64 (for rsync.o.o)
ip6:2a07:de40:401::/64 (for provo mirror, status2.o.o, maybe others).
Updated by pjessen over 2 years ago
pjessen wrote:
I am currently running a small experiment (only delivering outbound mails via ipv4), if that works,
The last bounce by Google was at 10:45 CET. I amended the postfix config an anna at 11:23 CET. Since then, all outgoing mails have been delivered to IPv4 addresses, without a single bounce.
Updated by pjessen over 2 years ago
This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org
Summary: 4.29% of all messages to google were bounced
Logfile is /var/log/mail-20220523.xz
Number of messages attempted = 12916
Number of messages successfully sent = 5712
Number of messages deferred (quota) = 332
Number of messages deferred (rate) = 0
Number of messages deferred (other) = 0
Number of messages deferred (total) = 332
Number of messages bounced (quota) = 0
Number of messages bounced (nosuchuser) = 0
Number of messages bounced (userdsbled) = 0
Number of messages bounced (blocked) = 0
Number of messages bounced (spam) = 0
Number of messages bounced (reputation) = 0
Number of messages bounced (authen) = 555
Number of messages bounced (other) = 0
Number of messages bounced (total) = 555
Updated by pjessen over 2 years ago
pjessen wrote:
pjessen wrote:
I am currently running a small experiment (only delivering outbound mails via ipv4), if that works,
The last bounce by Google was at 10:45 CET. I amended the postfix config on anna at 11:23 CET. Since then, all outgoing mails have been delivered to IPv4 addresses, without a single bounce.
It has only been a bit more than 22 hours, but still no bounces since 10:45 CET yesterday. The 4.29% bounced is also a significant improvement. I think that is a very clear indication that there is an issue with IPv6.
Updated by pjessen over 2 years ago
Following Lars' suggestion "don't play with production", I have set up a test scheme using opensuse.mx. It has a separate SPF record:
ip6:2001:67c:2178:8::/64 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48
^^^^^^
I force all outbound mail from opensuse.mx to be sent via ipv6. So far I have sent myself about 100 test mails, no bounces.
Updated by pjessen over 2 years ago
- Status changed from In Progress to Feedback
- Assignee changed from opensuse-admin to pjessen
- % Done changed from 0 to 40
pjessen wrote:
So far I have sent myself about 100 test mails, no bounces.
Plus another 100 and 100 to varying addresses (that volunteered to help out). All went fine.
After updating the SPF record to use 2001:67c:2178:8::/64, to revert to normal sending, I'll comment out 'smtp_address_preference'.
This will at least help our subscribers with gmail and googlemail, but whether it will help with $SUBJ, I don't know.
What about the other three ranges -
- 2a01:138:a004::/48 - is this used for anything else than rsync.o.o? I see 2a01:138:a004::202 being used, so restricting the SPF record to '2a01:138:a004::202/64' ought to be okay, especially as rsync.o.o only sends to us, not to Google :-)
- 2a07:de40:401::/48 - this is being used in provo, provo-mirror and status2. Again, 2a07:de40:401::/64 seems reasonable, especially as they only send to us, not to Google :-)
- 2620:113:8044::/48 - is this actually being used to send mails from opensuse.org?
Updated by pjessen over 2 years ago
pjessen wrote:
- 2620:113:8044::/48 - is this actually being used to send mails from opensuse.org?
It is not mentioned here: https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Network
Updated by pjessen over 2 years ago
- Status changed from Feedback to In Progress
- % Done changed from 40 to 50
at 18:30 UTC I
- changed the IPv6 entries in the SPF record to ip6:2001:67c:2178:8::/64 ip6:2a01:138:a004::/64 ip6:2a07:de40:401::/64
- anna: re-enabled delivery via IPv6
Many deliveries already went through fine, but that's no proof. I'll be keeping an eye on it for the next couple of hours, and check the bounce numbers tomorrow.
Updated by pjessen over 2 years ago
Oh well. There goes that theory.
Between 1830 and 1839, 131 mails to gmail addresses bounced. I have reverted to delivery by IPv4 only.
When it worked so perfectly fine for opensuse.mx, I have to wonder if it is DMARC related, opensuse.mx does not have a DMARC record. I have added a DMARC record for opensuse.mx.
Updated by pjessen over 2 years ago
pjessen wrote:
When it worked so perfectly fine for opensuse.mx, I have to wonder if it is DMARC related, opensuse.mx does not have a DMARC record. I have added a DMARC record for opensuse.mx.
I had hoped mails from opensuse.mx would also be bounced, but they were not. Another wild theory - the "reputation" of opensuse.mx is fine, whereas the "reputation" of opensuse.org is very poor ?
Updated by pjessen over 2 years ago
Well, I am leaving anna to send outbound mails over ipv4 only. This is very crude measure, and it affects all recipient addresses. I have also amended the postfix config on mx{12}.o.o to do the same, deliveries only over ipv4. I cannot explain why ipv4 vs ipv6 should make any difference.
Updated by pjessen over 2 years ago
This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org
Summary: 0.78% of all messages to google were bounced
Logfile is /var/log/mail-20220524.xz
Number of messages attempted = 21757
Number of messages successfully sent = 16107
Number of messages deferred (quota) = 1087
Number of messages deferred (rate) = 196
Number of messages deferred (other) = 18446744073709551426
Number of messages deferred (total) = 1093
Number of messages bounced (quota) = 11
Number of messages bounced (nosuchuser) = 3
Number of messages bounced (userdsbled) = 1
Number of messages bounced (blocked) = 1
Number of messages bounced (spam) = 0
Number of messages bounced (reputation) = 0
Number of messages bounced (authen) = 154
Number of messages bounced (other) = 0
Number of messages bounced (total) = 170
Updated by pjessen over 2 years ago
I may have just found out why ipv4 vs ipv6 seems to making a difference here. Having limited delivery to ipv4, we just don't get any bounces from Google. Probably because:
# dig lists.opensuse.org txt
;; ANSWER SECTION:
lists.opensuse.org. 1800 IN TXT "v=spf1 mx a:proxy-nue1.opensuse.org a:proxy-nue2.opensuse.org ?all"
# host proxy-nue1.opensuse.org
proxy-nue1.opensuse.org has address 195.135.221.145
# host proxy-nue2.opensuse.org
proxy-nue2.opensuse.org has address 195.135.221.139
So, we basically do not permit outbound list mails over IPv6 ...... aaaaaarg. I can't believe I did not see that earlier. It probably has no impact on $SUBJ though.
Updated by pjessen over 2 years ago
Earlier, I amended the SPF record for lists.o.o, adding ip6:2001:67c:2178:8::/64 and making it -all. Mailman is the only one sending anything with sender lists.o.o, and mailman forwards over relay.infra.opensuse.org only. For about three hours, everything to gmail and googlemail has been sent over ipv6, nothing has bounced due to "authentication". Yippee!
Updated by pjessen over 2 years ago
- Related to tickets #111536: Informing users who have been bounced by Google due to our misconfigured spf record for lists.o.o added
Updated by pjessen over 2 years ago
To return to $SUBJ, I have been testing with opensuse.mx. At first, I configured the SPF record to have '-all'. Sending a mail from per@opensuse.mx from my own system was correctly bounced by Google:
550-5.7.26 This message fails to pass SPF checks for an SPF record with a hard
550-5.7.26 failpolicy. To best protect our users from spam and phishing, the
550-5.7.26 message has been blocked. Please visit
550-5.7.26 https://support.google.com/mail/answer/81126#authentication for more
550 5.7.26 information.
"spf record with hard failpolicy" - sounds absolutely correct.
Then I changed the SPF record to have '?all' and sent about 100 test-mails from per@opensuse.mx to my gmail address. They are still arriving, so far none have bounced.
Currently, opensuse.mx has no DMARC policy.
Updated by cboltz over 2 years ago
pjessen wrote:
Earlier, I amended the SPF record for
lists.o.o, adding ip6:2001:67c:2178:8::/64 and making it-all. Mailman is the only one sending anything with senderlists.o.o, and mailman forwards overrelay.infra.opensuse.orgonly. For about three hours, everything to gmail and googlemail has been sent over ipv6, nothing has bounced due to "authentication". Yippee!
Nice catch!
However, I don't really like -all because it breaks forwarding. Maybe you could use ~all (softfail) instead to give people who forward their mailinglist mails to another mailbox a little chance?
Updated by pjessen over 2 years ago
This update was automatically generated by /usr/local/bin/google-traffic on anna.infra.opensuse.org
Summary: 0.02% of all messages to google were bounced
Logfile is /var/log/mail-20220525.xz
Number of messages attempted = 42022
Number of messages successfully sent = 29770
Number of messages deferred (quota) = 718
Number of messages deferred (rate) = 270
Number of messages deferred (other) = 18446744073709551368
Number of messages deferred (total) = 740
Number of messages bounced (quota) = 10
Number of messages bounced (nosuchuser) = 0
Number of messages bounced (userdsbled) = 2
Number of messages bounced (blocked) = 0
Number of messages bounced (spam) = 0
Number of messages bounced (reputation) = 0
Number of messages bounced (authen) = 0
Number of messages bounced (other) = 0
Number of messages bounced (total) = 12
Updated by pjessen over 2 years ago
cboltz wrote:
However, I don't really like
-allbecause it breaks forwarding. Maybe you could use~all(softfail) instead to give people who forward their mailinglist mails to another mailbox a little chance?
In principle, any forwarder ought to use SRS, but yeah why not, I suppose it won't hurt us :-)
Updated by pjessen over 2 years ago
pjessen wrote:
To return to $SUBJ, I have been testing with opensuse.mx.
I changed the SPF record to have '?all' and sent about 100 test-mails from per@opensuse.mx to my gmail address. They are still arriving, so far none have bounced.
Currently, opensuse.mx has no DMARC policy.
Comparing opensuse.mx and opensuse.org setup:
o.mx spf: v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 ip6:2a01:138:a004::/64 ip6:2a07:de40:401::/64 mx ?all
o.org spf: v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 ip6:2a01:138:a004::/64 ip6:2a07:de40:401::/64 mx ?all
o.mx dmarc: none
o.org dmarc: v=DMARC1; p=none; pct=100; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m
o.mx usage: nothing much at all.
o.org usage: everything.
For testing, I have to remove the DMARC record for opensuse.org, for maybe 24 hours. I hope that is okay. We have a holiday coming up.
Updated by pjessen over 2 years ago
pjessen wrote:
For testing, I have to remove the DMARC record for opensuse.org, for maybe 24 hours. I hope that is okay. We have a holiday coming up.
Well, that made no difference. Right now, with the exact same setup, the same mail sent from opensuse.mx and opensuse.org (from my home setup) is accepted and rejected, respectively.
I have re-instated the DMARC record for opensuse.org.
Updated by pjessen about 2 years ago
- Related to tickets #116938: sending o.o mails to gmail account not possible added
Updated by pjessen about 2 years ago
pjessen wrote:
cboltz wrote:
BTW: Ubuntu (who also provide mail aliases for their members) have
v=spf1 include:canonical.com +allNice example. I suggest we follow it.
I thought I would check again - the SPF record for ubuntu.com remains the same (see above), and interestingly they don't publish any DMARC nor DKIM records. It seems to me they ought to have the same issues we are having.
For my testing with opensuse.mx, I have updated the SPF record to say "+all". I may take a closer look when I have some time.
Updated by pjessen about 2 years ago
pjessen wrote:
For my testing with
opensuse.mx, I have updated the SPF record to say "+all". I may take a closer look when I have some time.
opensuse.mx has no dmarc nor dkim record.
spf record is v=spf1 include:_spf.opensuse.org +all
_spf.opensuse.org is v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 ip6:2a01:138:a004::/64 ip6:2a07:de40:401::/64 mx ?all
With that config, test mails from per@opensuse.mx to my own gmail address bounced, "suspected spam".
Updated by pjessen about 2 years ago
New try:
_spf.opensuse.mx = v=spf1 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 mx -all
This is quite strict, and will intentionally fail anything not coming from an openSUSE IP range. However,
with the SPF record = v=spf1 include:_spf.opensuse.mx +all, I am hoping the '+all' will override the previous -all
Updated by pjessen about 2 years ago
pjessen wrote:
New try:
_spf.opensuse.mx =v=spf1 ip4:195.135.220.0/23 ip6:2001:67c:2178:8::/64 mx -all
This is quite strict, and will intentionally fail anything not coming from an openSUSE IP range. However,
with the SPF record =v=spf1 include:_spf.opensuse.mx +all, I am hoping the '+all' will override the previous-all
With the config above, sending a mail from per@opensuse.mx to my gmail address still results in the following bounce:
<redacted@gmail.com>: host gmail-smtp-in.l.google.com[74.125.140.27] said:
550-5.7.1 [88.198.198.124 12] Our system has detected that this message is
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail,
550-5.7.1 this message has been blocked. Please visit
550-5.7.1 https://support.google.com/mail/?p=UnsolicitedMessageError
550 5.7.1 for more information.
Updated by pjessen about 2 years ago
I would like if someone would review my setup for opensuse.mx, just to make sure I am not missing something important.
The objective is to enable anyone to send emails with their opensuse alias address, from whereever they are. The +all fall-back SPF policy should enable this, but Google does not accept it.
Currently, Google rejects mails from opensuse.org and opensuse.mx, apparently ignoring the +all fall-back SPF policy.
Updated by DocB over 1 year ago
pjessen wrote:
Currently, Google rejects mails from opensuse.org and opensuse.mx, apparently ignoring the
+allfall-back SPF policy.
looks like this is stil the case - and I guess google gives a shit about '+all'?
Updated by pjessen over 1 year ago
DocB wrote:
pjessen wrote:
Currently, Google rejects mails from opensuse.org and opensuse.mx, apparently ignoring the
+allfall-back SPF policy.looks like this is stil the case - and I guess google gives a shit about '+all'?
I would be a lot less diplomatic about it ... yes, I wonder if Google actually treats "+all" as even worse.
To be honest, it is so frustrating, I have essentially given up on it.