Project

General

Profile

tickets #109025

gmail rejects mails from cboltz@o.o - SPF/DKIM/DMARC issue

Added by cboltz about 2 months ago. Updated about 1 hour ago.

Status:
In Progress
Priority:
Normal
Assignee:
opensuse-admin
Category:
Email
Target version:
-
Start date:
2022-03-27
Due date:
% Done:

0%

Estimated time:

Description

I tried to send a mail with cboltz@o.o as sender (using my own mail server), but gmail rejected it:

<REDACTED@gmail.com>: host gmail-smtp-in.l.google.com[142.250.153.26] said:
    550-5.7.26 This message does not have authentication information or fails
    to 550-5.7.26 pass authentication checks. To best protect our users from
    spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
    https://support.google.com/mail/answer/81126#authentication for more 550
    5.7.26 information.

Sending a mail with nearly the same content and using the same server, but using a mail address @cboltz.de as sender worked without problems.

This looks like a problem with the SPF and/or DKIM and/or DMARC settings for opensuse.org. A test mail to https://www.kbxscore.com with cboltz@o.o as sender gets reported as

DKIM:  FAIL     No DKIM authentication signature found in this email message.
SPF:   SOFTFAIL (domain owner discourages use of this host) identity=mailfrom; client-ip=88.99.101.17; helo=mail.cboltz.de; envelope-from=REDACTED@opensuse.org; receiver=REDACTED@kbxscore.com
DMARC: FAIL      Uh-oh! Message did not pass DMARC checks. No DKIM signature found for opensuse.org. DMARC policy for opensuse.org: p=none;​ pct=100;​ rua=mailto:admin-auto@opensuse.org!5m;​ ruf=mailto:admin-auto@opensuse.org!5m 

The missing DKIM signature and the SPF softfail are not surprising - since we don't offer a SMTP server for our members to send out mails as $member@o.o, I have to use my own server (which of course can't create a DKIM signature for o.o, and also is outside of the IP range in the SPF entry).

Looking at the reports on admin-auto, it seems I'm not the only one who suffers from this problem.

I'm afraid there are only two ways how we can fix that:

  • stop using SPF, DKIM and DMARC for opensuse.org - or -
  • provide a SMTP server the members can use

History

#1 Updated by cboltz about 2 months ago

  • Private changed from Yes to No

#2 Updated by lrupp about 2 months ago

cboltz wrote:

I'm afraid there are only two ways how we can fix that:

  • stop using SPF, DKIM and DMARC for opensuse.org - or -
  • provide a SMTP server the members can use

That Google (or any other provier) rejects Emails from dynamic IP addresses is not really new - and even wanted to avoid SPAM.

As some providers are meanwhile not accepting Emails from Domains without DKIM support (or give them a SPAM rating), I suggest that we might consider the 2nd option as a possible solution. DKIM is also the interim step towards DMARC, which is already requested for openSUSE since a while now.

#3 Updated by lrupp about 2 months ago

  • Status changed from New to Feedback
  • Assignee changed from lrupp to cboltz

IMHO this ticket should be discussed a bit wider. I'm just not sure if it's enough to start a discussion on the heroes mailing list, on the project mailing list or at code.opensuse.org.

=> Christian: what do you think?

#4 Updated by cboltz about 2 months ago

I think we should start on the heroes ML (or in the meeting next week) and decide which way we want to choose.

If we decide to provide a SMTP server for sending mails as $member@o.o, this is of course something to announce on the project ML as soon as it's ready.

#5 Updated by cboltz about 2 months ago

Oh, forgot to mention - no dynamic IP involved. My mail server lives at Hetzner with a static IP, reverse DNS etc.

#6 Updated by pjessen about 2 months ago

cboltz wrote:

I tried to send a mail with cboltz@o.o as sender (using my own mail server), but gmail rejected it:

FWIW, I have just now tried the same, to my own gmail address - no problems.

Looking at the reports on admin-auto, it seems I'm not the only one who suffers from this problem.

You mean the dmarc reports?

I'm afraid there are only two ways how we can fix that:

  1. stop using SPF, DKIM and DMARC for opensuse.org - or -
  2. provide a SMTP server the members can use

I vote yes to (1) and no to (2).

(1) I don't believe we are gaining anything.
(2) in principle easy, in practice it's a lot of effort.

#7 Updated by pjessen about 2 months ago

lrupp wrote:

As some providers are meanwhile not accepting Emails from Domains without DKIM support (or give them a SPAM rating),
I suggest that we might consider the 2nd option as a possible solution. DKIM is also the interim step towards DMARC,
which is already requested for openSUSE since a while now.

If whoever is requesting it is also funding the effort, I'll be happy to make an offer for the contract. (no smiley).

#8 Updated by cboltz about 1 month ago

  • Status changed from Feedback to New
  • Assignee changed from cboltz to lrupp

We discussed this in the meeting, and agreed that we are not keen on setting up a mail relay for the Members ;-)

This also means we'll need to relax or drop the SPF etc. entries so that our Members can continue to send mails as $username@o.o. Lars, can you please do that?

#9 Updated by pjessen about 1 month ago

cboltz wrote:

We discussed this in the meeting, and agreed that we are not keen on setting up a mail relay for the Members ;-)
This also means we'll need to relax or drop the SPF etc. entries so that our Members can continue to send mails as $username@o.o. Lars, can you please do that?

I would be tempted to set up SPF to explicitly allow from anywhere. Or anywhere with a proper DNS setup (reverse->forward->reverse). I'm not sure if SPF even has such an option :-)

#10 Updated by pjessen 29 days ago

I also just had a mail rejected by google:

<someone@gmail.com>: host gmail-smtp-in.l.google.com[64.233.184.26] said:
    550-5.7.26 This message does not have authentication information or fails
    to 550-5.7.26 pass authentication checks. To best protect our users from
    spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
    https://support.google.com/mail/answer/81126#authentication for more 550
    5.7.26 information. 17-20020a05600c029100b0038ecfc8e76fsi7384276wmk.33 -
    gsmtp (in reply to end of DATA command)

Sent from my opensuse.org address, now resending from jessen.ch.

#11 Updated by lrupp 27 days ago

DMARC

I just checked the DMARC DNS entries:

v=DMARC1; p=none; pct=100; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m

^ Please note the "Policy=none" section. DMARC was already disabled. Only reports where sent.
I enhanced this a bit further now and set:

v=DMARC1; p=none; pct=0; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m

Which also tells the receiving side that 0 (zero) percent of the Emails should fall under the 'none' category (which is already the lowest one). So whatever brings Google to reject your Emails seems not to be related to DMARC.

SPF

Regarding SPF we have:

v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~all

According to Google's documentation, the ~all should result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets. We could switch here to -all to completely disable SPF (like DMARC) as well, but I wonder why these kind of rejects just happen now, while we have the SPF records since (at least) 2021-09-17 (which is our oldest backup). I assume, we have this SPF setting even longer.

DKIM

Regarding DKIM: this is IMHO not implemented in any openSUSE domain.

Any thoughts?

#12 Updated by cboltz 26 days ago

lrupp wrote:

SPF

Regarding SPF we have:

v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~all

According to Google's documentation, the ~all should result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets. We could switch here to -all to completely disable SPF (like DMARC) as well,

Correct me if I'm wrong, but -all would mean to reject mails from other servers, which would make sending mails as cboltz@o.o from my own mail server impossible.

IMHO ?all or even +all would be a better choice.

but I wonder why these kind of rejects just happen now, while we have the SPF records since (at least) 2021-09-17 (which is our oldest backup). I assume, we have this SPF setting even longer.

The most boring explanation I can offer is: I don't use my @o.o address too often, so it might be that I didn't send a mail with the @o.o sender to gmail for a year or two.

#13 Updated by pjessen 25 days ago

lrupp wrote:

## DMARC

I just checked the DMARC DNS entries:

I would be tempted not to publish any DMARC policy at all, but there might be some best-practices information on that.

## SPF

Regarding SPF we have:

v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx ~all

According to Google's documentation, the ~all should result in Google marking Emails as suspicious, if they are NOT from any of the listed subnets.

Yes, a softfail is suspicious, but not normally a real problem.

We could switch here to -all to completely disable SPF (like DMARC) as well,

I think the ~all is okay - I see other email-alias providers (ieee.org, acm.org) using the same.
Or as Christian suggested, simply "v=spf1 +all" - although that is the same as not publishing any SPF info.

#14 Updated by cboltz 22 days ago

~ means "softfail" and makes it likely to get the mail tagged as spam (or even rejected, like the mail that triggered this ticket), and that's not what should happen to mails sent by our Members. Therefore I'd really prefer ?all (= "neutral") or +all (= "pass"). Since we expect our Members to use their own mailservers, +all would even be "technically correct" ;-)

BTW: Ubuntu (who also provide mail aliases for their members) have v=spf1 include:canonical.com +all

#15 Updated by pjessen 22 days ago

cboltz wrote:

~ means "softfail" and makes it likely to get the mail tagged as spam (or even rejected, like the mail that triggered this ticket), and that's not what should happen to mails sent by our Members.

+1

Therefore I'd really prefer ?all (= "neutral") or +all (= "pass"). Since we expect our Members to use their own mailservers, +all would even be "technically correct" ;-)

+1

BTW: Ubuntu (who also provide mail aliases for their members) have v=spf1 include:canonical.com +all

Nice example. I suggest we follow it.

#16 Updated by pjessen 22 days ago

FWIW, I had two emails to gmail rejected today, same "explanation".

#17 Updated by lrupp 19 days ago

  • Status changed from New to Workable
  • Assignee changed from lrupp to opensuse-admin

1) DKIM and DMARC are out of this discussion. We are only talking about the SPF entries here.

2) SPF
In the past, opensuse.org often enough ended up on one of the public blacklists, as we allowed everyone (including Spammers) to submit his @opensuse.org EMail from everywhere. Since we added the ~all SPF record, opensuse.org was not ending on one single blacklist any more.

Therefor, I vote for keeping the entry as it is. But that's just me (see below).

3) General

I could think about a specific (additional) SMTP server, which could be used by our openSUSE-members for sending their opensuse.org-Emails by authenticating to this server. But I don't see me here as the one who should make the final decision, so leaving it up to "the crowd" ;-)

#18 Updated by pjessen 18 days ago

lrupp wrote:

1) DKIM and DMARC are out of this discussion. We are only talking about the SPF entries here.

Why are they out of this disucssion? What is new here is the DMARC policy, I don't recall any rejects by google before we published a DMARC policy ?
My gut feeling say it is the above that is causing the problems. Granted, it could be google having changed policies too, but as our SPF entry did not change, I don't think the problem is SPF.

3) General
I could think about a specific (additional) SMTP server, which could be used by our openSUSE-members for sending their opensuse.org-Emails by authenticating to this server. But I don't see me here as the one who should make the final decision, so leaving it up to "the crowd" ;-)

It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.

Maybe we could disable our DMARC policy (i.e. remove it) for a month, and run some tests to see if we are still getting bounces?

#19 Updated by bmwiedemann 18 days ago

It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.

I think, the SMTP server would only allow accounts from openSUSE members that send with their assigned @opensuse.org alias. So random spammers that just signed up for a new account would not be allowed to relay their mails.

And without SPF/DMARC, spammers did not even need to bother with an account, but could just send from anywhere and there was no way for receivers to tell that this is not us doing it.

#20 Updated by pjessen 18 days ago

bmwiedemann wrote:

It seems to me the challenges / risks would remain the same, namely how to prevent abuse and blacklisting.

I think, the SMTP server would only allow accounts from openSUSE members that send with their assigned @opensuse.org alias.

Yes certainly, but that is not the point. Maybe see my posting to the heroes list from the beginning of April:
https://lists.opensuse.org/archives/list/heroes@lists.opensuse.org/thread/4A7ETASHCJB5Z64LSLTTXVFMW7RRP5LD/

Here is the not so easy part:

With hundreds of user credentials spread around the world in uncontrolled/insecure locations, the risk of one getting compromised
or "borrowed" cannot be ignored.  It is a spammer's wet dream - access to a mailserver with DMARC authentication.

If our outgoing mailserver is seen to be spamming, it won't be long before no one can send anything at all.  To mitigate this risk, we will
need rate controls and checks on userids being used from multiple locations. A lot less easy, but still it can be done.

The question is - is it worth it ?

And without SPF/DMARC, spammers did not even need to bother with an account, but could just send from anywhere
and there was no way for receivers to tell that this is not us doing it.

It isn't us :-) That is the current situation and that is what we want, in my opinion.

#21 Updated by bmwiedemann 14 days ago

It is a spammer's wet dream - access to a mailserver with DMARC authentication.

It is pretty common these days for spammers to register their own domain and have valid DKIM+DMARC for it. It is probably part of hosters' standard deployment.
E.g. I got one from @chimeicorpplc.pw

So not really that bad. If it happens, the user gets assigned a new PW.

Alternatively we hand out SSL client certs and if one leaks, it gets added to the CRL.

#22 Updated by pjessen 13 days ago

bmwiedemann wrote:

It is a spammer's wet dream - access to a mailserver with DMARC authentication.

It is pretty common these days for spammers to register their own domain and have valid DKIM+DMARC for it. It is probably part of hosters' standard deployment.
E.g. I got one from @chimeicorpplc.pw

So not really that bad. If it happens, the user gets assigned a new PW.

I guess we will discover it when our mailserver has been blacklisted.

Alternatively we hand out SSL client certs and if one leaks, it gets added to the CRL.

I have yet to hear the business case for all of this effort. It seems like much ado about nothing. Anyway, enough from me, I am opposed to it.

#23 Updated by cboltz 11 days ago

As just discussed and decided in the Heroes meeting, I changed the SPF entry to +all. In theory this should make things better ;-)

#24 Updated by pjessen 6 days ago

I just had two emails rejected by google:

<helpdesk@clarkson.edu>: host aspmx.l.google.com[173.194.76.27] said: 550-5.7.1
    [88.198.198.124      12] Our system has detected that this message is
    550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to
    Gmail, 550-5.7.1 this message has been blocked. Please visit 550-5.7.1
    https://support.google.com/mail/?p=UnsolicitedMessageError 550 5.7.1  for
    more information. o3-20020a05600c2e0300b0039446b4a065si1496380wmf.30 -
    gsmtp (in reply to end of DATA command)
<medwinz@gmail.com>: host gmail-smtp-in.l.google.com[173.194.76.26] said:

    550-5.7.26 This message does not have authentication information or fails
    to 550-5.7.26 pass authentication checks. To best protect our users from
    spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
    https://support.google.com/mail/answer/81126#authentication for more 550
    5.7.26 information. k5-20020a056000004500b00207a5488326si12156215wrx.446 -
    gsmtp (in reply to end of DATA command)

I'll try sending them both from my private address, per@jessen.ch

#25 Updated by pjessen 6 days ago

  • Status changed from Workable to Feedback

pjessen wrote:

I'll try sending them both from my private address, per@jessen.ch

They both went through. Looking up one of the URLs suggested in the rejection message, and google suggests signing up for "Postmaster Tools".
This will require amending the DNS setup, adding a verification record - does anyone see a problem if I go ahead and do that, using my personal Google account?

#26 Updated by pjessen 6 days ago

  • Status changed from Feedback to In Progress

pjessen wrote:

Looking up one of the URLs suggested in the rejection message, and google suggests signing up for "Postmaster Tools".
This will require amending the DNS setup, adding a verification record - does anyone see a problem if I go ahead and do that, using my personal Google account?

I have added the google verification TXT record.

#27 Updated by pjessen 6 days ago

pjessen wrote:

Looking up one of the URLs suggested in the rejection message, and google suggests signing up for "Postmaster Tools".

In "Postmaster Tools" there is a dashboard with various bits of information, reputation etc. I don't immediately see that anything is wrong. I can add other users, let me know (offline) which addresses to add (I don't know if they have to be google accounts, but probably not).

#28 Updated by pjessen 5 days ago

Another bounce:

<ftpadmin@g.unicamp.br>: host aspmx.l.google.com[66.102.1.27] said: 550-5.7.1
    [88.198.172.124      12] Our system has detected that this message is
    550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to
    Gmail, 550-5.7.1 this message has been blocked. Please visit 550-5.7.1
    https://support.google.com/mail/?p=UnsolicitedMessageError 550 5.7.1  for
    more information. y3-20020a5d6203000000b00207a6a693dcsi1693973wru.277 -
    gsmtp (in reply to end of DATA command)

I'll resend from my private address.

And another one later:

<apoio@ccuec.unicamp.br>: host aspmx.l.google.com[64.233.167.27] said:
    550-5.7.1 [88.198.172.124      12] Our system has detected that this
    message is 550-5.7.1 likely unsolicited mail. To reduce the amount of spam
    sent to Gmail, 550-5.7.1 this message has been blocked. Please visit
    550-5.7.1  https://support.google.com/mail/?p=UnsolicitedMessageError 550
    5.7.1  for more information.
    a6-20020a05600c348600b003947b59dfeesi222965wmq.8 - gsmtp (in reply to end
    of DATA command)

#29 Updated by pjessen 4 days ago

It may be a bit premature, but it seems to me we can conclude that our +all SPF policy doesn't work. When I send from @jessen.ch, there is no problem. (jessen.ch has an SPF record, but no DMARC policy).

I have added my own mailserver mail.hostsuisse.com to the 'opensuse.org' SPF record, just as an experiment.
It has the following two addresses, and only mails from 'per@opensuse.org' will be sent.

185.85.248.6
2a03:7520:4c68::6

#30 Updated by pjessen 4 days ago

pjessen wrote:

I have added my own mailserver mail.hostsuisse.com to the 'opensuse.org' SPF record, just as an experiment.

On test#1, it worked! This is obviously not a solution, but seems to indicate that +all does not quite work.

#31 Updated by pjessen 4 days ago

pjessen wrote:

pjessen wrote:

I have added my own mailserver mail.hostsuisse.com to the 'opensuse.org' SPF record, just as an experiment.

On test#1, it worked! This is obviously not a solution, but seems to indicate that +all does not quite work.

I did another few tests, all worked. I have now reduced the SPF record :

from v=spf1 ip4:91.193.113.64/27 ip4:143.186.213.0/24 ip4:147.2.0.0/16 ip4:149.44.0.0/16 ip4:195.135.220.0/23 ip6:2001:67c:2178::/48 ip6:2620:113:8044::/48 ip6:2a01:138:a004::/48 ip6:2a07:de40:401::/48 mx +all

to v=spf1 all

So far that has worked too.

#32 Updated by pjessen 3 days ago

Aaaaaarg:

<justin.w.xd@gmail.com>: host gmail-smtp-in.l.google.com[173.194.76.27] said:
    550-5.7.26 This message does not have authentication information or fails
    to 550-5.7.26 pass authentication checks. To best protect our users from
    spam, the 550-5.7.26 message has been blocked. Please visit 550-5.7.26
    https://support.google.com/mail/answer/81126#authentication for more 550
    5.7.26 information. g5-20020adfd1e5000000b0020ad1f6b8cfsi1081934wrd.635 -
    gsmtp (in reply to end of DATA command)

#33 Updated by pjessen 3 days ago

I had our DMARC policy checked at https://www.dmarcanalyzer.com :

DNS record: v=DMARC1; p=none; pct=0; rua=mailto:admin-auto@opensuse.org!5m; ruf=mailto:admin-auto@opensuse.org!5m

Problems:

We've noticed the following problems with your DMARC records .
Error : Invalid "pct" value given in combination with "none" policy ("0" given, only 100 is supported).
Warning : There is a size limit present in your "rua" tag: mailto:admin-auto@opensuse.org!5m. It is advised to avoid using this for now as it is not supported by all ISP's and could lead to missing reports.
Warning : There is a size limit present in your "ruf" tag: mailto:admin-auto@opensuse.org!5m. It is advised to avoid using this for now as it is not supported by all ISP's and could lead to missing reports.
Info : We have found a custom address in your DMARC record. Please make sure to forward the incoming reports to your custom address to see the statistics in the tool.

The report about the "pct" value sounds like it could be a real issue, I'll change it to 100.

#34 Updated by pjessen about 21 hours ago

I guess this is progress:

<justin.w.xd@gmail.com>: host alt1.gmail-smtp-in.l.google.com[142.250.153.27]
    said: 451-4.7.24 [88.198.198.124      15] The SPF record of the sending
    domain has one 451-4.7.24 or more suspicious entries. To protect our users
    from spam, mail 451-4.7.24 sent from your IP address has been temporarily
    rate limited. Please 451-4.7.24 visit 451-4.7.24
    https://support.google.com/mail/answer/81126#authentication for more 451
    4.7.24 information. qa18-20020a170907869200b006f4405ff418si5420573ejc.453 -
    gsmtp (in reply to end of DATA command)

I can only presume that "one or more suspicious entries" in the SPF record is our "all". The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable". I have changed our SPF record to say ?all.

#35 Updated by lrupp about 1 hour ago

pjessen wrote:

I can only presume that "one or more suspicious entries" in the SPF record is our "all".
The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable".
I have changed our SPF record to say ?all.

I have to admit that I'm not a big fan of playing around with productive systems all the time: can I suggest that you use one of the alternative DNS domains for your SPAM testing?

We have enough of them:
opensuse-project.com

opensuse-project.de

opensuse-project.net

opensuse-project.org

opensuse.asia

opensuse.co.in

opensuse.co

opensuse.com.br

opensuse.com.es

opensuse.com.mx

opensuse.com

opensuse.de

opensuse.eu

opensuse.fr

opensuse.gen.tr

opensuse.jp

opensuse.kr

opensuse.me

opensuse.mu
opensuse.mx
opensuse.net
opensuse.org.cn
opensuse.org
opensuse.pk

Regards,
Lars

#36 Updated by pjessen about 1 hour ago

lrupp wrote:

pjessen wrote:

I can only presume that "one or more suspicious entries" in the SPF record is our "all".
The mail will have been retried a number of times over 24 hours, after which my mailserver said "undeliverable".
I have changed our SPF record to say ?all.

I have to admit that I'm not a big fan of playing around with productive systems all the time: can I suggest that you use one of the alternative DNS domains for your SPAM testing?

Yeah, that is probably not a bad idea, although we should perhaps have done with the DMARC setup too :-)

Fwiw, I have now had two mails successfully delivered to Google, one to a hosted customer (clarkson.edu) and one to a plain gmail user.

Also available in: Atom PDF