Project

General

Profile

Actions

tickets #72133

closed

openSUSE mailing lists break domains like suse.com that use DMARC/DKIM

Added by esujskaja over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Email
Target version:
-
Start date:
2020-09-30
Due date:
% Done:

100%

Estimated time:

Description

I would propose admin@opensuse.orgadmin@opensuse.org for beginning, so I added them.

I also dared to add Per (hope he don’t mind) – just because I know he was very much in openSUSE mail server related work this summer – Per, would you please help us to find the right contacts?

Many thanks!
Z

From: Michalis Kamprianis michalis.kamprianis@suse.com
Sent: Tuesday, September 29, 2020 10:06 AM
To: Gerald Pfeifer gp@suse.com
Cc: Evženie Šujskaja evzenie.sujskaja@suse.com
Subject: need to contact opensuse.org mail admins
Importance: High

Hi Gerald
We are implementing DMARC on suse.com and at the moment, opensuse.org mailing lists are among the ones that fail to treat DKIM right. Of all the mails that we would reject if we had DMARC on, 90% would be from opensuse.org

I tried to reach out to heroes@lists.opensuse.orgheroes@lists.opensuse.org but I couldn't get anywhere. I also know that Lars is on vacation. Do you have any other email I can use to get in touch with whoever manages the opensuse.org mailing lists?

Thanks

Michalis Kamprianis
IT Security Manager
SUSE LINUX, s. r. o.
Corso II
Křižíkova 148/34
18600 Praha 8
Tel: +420 702 133 218


Files

Screenshot_20201002_132709.png (38.3 KB) Screenshot_20201002_132709.png DKIM failure mkamprianis, 2020-10-02 11:30
Screenshot_20201002_133302.png (41.7 KB) Screenshot_20201002_133302.png DKIM success mkamprianis, 2020-10-02 11:34
Actions #1

Updated by pjessen over 3 years ago

  • Category set to Email
  • Assignee set to pjessen
  • Private changed from Yes to No
Actions #2

Updated by pjessen over 3 years ago

  • Status changed from New to Workable

The correct address for the 'heroes' list is heroes@opensuse.org.

I presume the DMARC issue is that mlmmj adds this text :

To unsubscribe ....
To contact the owner ....

to the email body, which screws up the checksum.

Removing it is easily done, "rm /var/spool/mlmmj/*/control/footer", but is whitelisting not an option?

Actions #3

Updated by mkamprianis over 3 years ago

pjessen wrote:

The correct address for the 'heroes' list is heroes@opensuse.org.

Thanks, I'll keep it in mind for next time.

I presume the DMARC issue is that mlmmj adds this text :

To unsubscribe ....
To contact the owner ....

to the email body, which screws up the checksum.

I have not seen the emails themselves, but that sounds plausible.

Removing it is easily done, "rm /var/spool/mlmmj/*/control/footer", but is whitelisting not an option?

Whitelisting is a client-side solution. We may whitelist whatever we want, but the rest of the world will need to whitelist too, or drop / quarantine the mails. That is why the mail has to be constructed properly; it's the safest solution to increase the deliverability of it.

I have requested DKIM admins in suse.de to add the l= DKIM field so that footer additions won't invalidate the DKIM header, but this is a slippery slope.
I would strongly prefer any non-body related parts (such as unsubscribe, owner etc) to be in the headers instead of the body.
Can we do that?

Actions #4

Updated by pjessen over 3 years ago

mkamprianis wrote:

pjessen wrote:

The correct address for the 'heroes' list is heroes@opensuse.org.

Thanks, I'll keep it in mind for next time.

I presume the DMARC issue is that mlmmj adds this text :

To unsubscribe ....
To contact the owner ....

to the email body, which screws up the checksum.

I have not seen the emails themselves, but that sounds plausible.

I expect also :

  • the [listname] that we prepend to the Subject line. This may be changed with the control/prefix option.
  • various headers that we delete before redistributing. This may be changed with the control/delheaders option.

Removing it is easily done, "rm /var/spool/mlmmj/*/control/footer", but is whitelisting not an option?

Whitelisting is a client-side solution. We may whitelist whatever we want, but the rest of the world will need to whitelist too, or drop / quarantine the mails. That is why the mail has to be constructed properly; it's the safest solution to increase the deliverability of it.

Well, so far it has not been an issue for us.

I have requested DKIM admins in suse.de to add the l= DKIM field so that footer additions won't invalidate the DKIM header, but this is a slippery slope.
I would strongly prefer any non-body related parts (such as unsubscribe, owner etc) to be in the headers instead of the body.
Can we do that?

Before we embark on some knee-jerk changes to how we run mlmmj, perhaps it is worth looking at other options - according to RFC6377 categories, our lists are virtually all "resending", so perhaps we could/should:

a) remove any existing DKIM headers
b) and/or add our own DKIM signature.

I feel option (a) is the least invasive and most easily implemented too.

Actions #5

Updated by pjessen over 3 years ago

  • Status changed from Workable to In Progress

pjessen wrote:

perhaps we could/should:

a) remove any existing DKIM headers
b) and/or add our own DKIM signature.

I feel option (a) is the least invasive and most easily implemented too.

As an experiment and because I am fairly certain it causes no harm, I have changed the config for opensuse, opensuse-support, opensuse-factory and opensuse-offtopic to delete the DKIM-Signature.

Actions #6

Updated by mkamprianis over 3 years ago

pjessen wrote:

pjessen wrote:

perhaps we could/should:

a) remove any existing DKIM headers
b) and/or add our own DKIM signature.

I feel option (a) is the least invasive and most easily implemented too.

As an experiment and because I am fairly certain it causes no harm, I have changed the config for opensuse, opensuse-factory and opensuse-offtopic to delete the DKIM-Signature.

This will be definitely blocked by us. DKIM is a mandatory part of DMARC. In order to protect the SUSE brand we are implementing DMARC.
If you send a mail from any address @suse.com without any DKIM signature, this will be dropped when we go live. This is the wrong way to handle DKIM.

  • You may add your DKIM (as you should anyway) for the envelope
  • You may stop sending with the user's domain and send from : whatever@opensuse.org

Both options are valid for DKIM. If you send with a From @suse.com, then mailservers will follow suse.com DMARC policy.

Actions #7

Updated by pjessen over 3 years ago

  • Status changed from In Progress to Feedback

mkamprianis wrote:

pjessen wrote:

pjessen wrote:

perhaps we could/should:

a) remove any existing DKIM headers
b) and/or add our own DKIM signature.

I feel option (a) is the least invasive and most easily implemented too.

As an experiment and because I am fairly certain it causes no harm, I have changed the config for opensuse, opensuse-factory and opensuse-offtopic to delete the DKIM-Signature.

This will be definitely blocked by us. DKIM is a mandatory part of DMARC. In order to protect the SUSE brand we are implementing DMARC.
If you send a mail from any address @suse.com without any DKIM signature, this will be dropped when we go live.

Okay, would you prefer the validation to fail over just removing the DKIM signature ?

Currently, I see our options as:

a) do nothing and keep re-sending invalid signatures
b) remove DKIM signature (to stop the validation failing)
c) you whitelist @suse.com mails resent by the opensuse mailing list server.
d) change our list processing (subject, headers, our information footers).
e) we add our own DKIM signature when we re-send messages

Isn't this DMARC enforcement going to cause problems on many other lists where people participate with suse.com addresses, for instance:

https://lists.freedesktop.org - all mailing lists (they prepend LISTNAME to the subject, and add a friendly footer, iow just like us)
https://www.redhat.com/mailman/listinfo/ - the same.
Apache Project mailing lists - the same.

Updated by mkamprianis over 3 years ago

pjessen wrote:

This will be definitely blocked by us. DKIM is a mandatory part of DMARC. In order to protect the SUSE brand we are implementing DMARC.
If you send a mail from any address @suse.com without any DKIM signature, this will be dropped when we go live.

Okay, would you prefer the validation to fail over just removing the DKIM signature ?

Currently, I see our options as:

a) do nothing and keep re-sending invalid signatures
b) remove DKIM signature (to stop the validation failing)
c) you whitelist @suse.com mails resent by the opensuse mailing list server.
d) change our list processing (subject, headers, our information footers).
e) we add our own DKIM signature when we re-send messages

(e) is irrelevant but of course more than welcome if you do it.
(c) will not solve the problem; even if we whitelist all opensuse.org mails to come through to suse.com (which we won't), our policy will still be enabled so everybody else will quarantine them. We may not see the problem, but the problem will be there.
(a) and (b): Think about a banking cheque. It may have a real signature, it may have a fake signature or it may have the signature part ripped off. Which one of the three can you cash?

Isn't this DMARC enforcement going to cause problems on many other lists where people participate with suse.com addresses, for instance:

https://lists.freedesktop.org - all mailing lists (they prepend LISTNAME to the subject, and add a friendly footer, iow just like us)
https://www.redhat.com/mailman/listinfo/ - the same.
Apache Project mailing lists - the same.

As you can see in one of the screenshots, in the last 7 days there were 135k emails with a @suse.com address. 72k of these were invalid. 60k of these were from opensuse.org
The 2nd domain sending the most invalid mails was rzone.de with 766. That is ~1% of what opensuse.org does. I will worry and handle all the other cases after we solve the major one which is opensuse.org

At the same time, you can see at the other screenshot that the most DMARC-compliant emails sent are from kernel.org mailing lists (~16k) followed by xenproject (~7k). Not all mailing lists are configured the same way.

Actions #9

Updated by pjessen over 3 years ago

mkamprianis wrote:

(c) will not solve the problem; even if we whitelist all opensuse.org mails to come through to suse.com (which we won't), our policy will still be enabled so everybody else will quarantine them. We may not see the problem, but the problem will be there.

Hmm, yes, true.

(a) and (b): Think about a banking cheque. It may have a real signature, it may have a fake signature or it may have the signature part ripped off. Which one of the three can you cash?

Cheques went out of fashion thirty years ago :-)

Isn't this DMARC enforcement going to cause problems on many other lists where people participate with suse.com addresses, for instance:

https://lists.freedesktop.org - all mailing lists (they prepend LISTNAME to the subject, and add a friendly footer, iow just like us)
https://www.redhat.com/mailman/listinfo/ - the same.
Apache Project mailing lists - the same.

As you can see in one of the screenshots, in the last 7 days there were 135k emails with a @suse.com address. 72k of these were invalid. 60k of these were from opensuse.org
The 2nd domain sending the most invalid mails was rzone.de with 766. That is ~1% of what opensuse.org does. I will worry and handle all the other cases after we solve the major one which is opensuse.org

Well, I am not keen on changing our 90-100 lists willy-nilly, especially when other major mailing list operators have not.
To test it, I think I'll make the changes for the heroes list and have someone write some test mails.

At the same time, you can see at the other screenshot that the most DMARC-compliant emails sent are from kernel.org mailing lists (~16k) followed by xenproject (~7k). Not all mailing lists are configured the same way.

That is precisely my point.

Actions #10

Updated by pjessen over 3 years ago

FYI, bugzilla sends emails from "bugzilla_noreply@suse.com" to the list "opensuse-bugs", but with no DKIM signature.
Once you start enforcing DMARC, I guess we (mx[12].o.o) will be the first to reject.
Also, I notice many people using "suse.de" addresses, but with no DKIM signature.

Actions #11

Updated by cboltz over 3 years ago

As an idea and question - can you please try a test mail to the opensuse-test@o.o mailinglist? opensuse-test is the first list we deliver via mailman3, so it would be good to know if mailman has similar problems, or if it just works[tm].

(If you aren't subscribed to opensuse-test, you can do so at https://lists.opensuse.org/manage/lists/test.lists.opensuse.org/ - that's the mailman3 web interface.)

Actions #12

Updated by pjessen over 3 years ago

To test it, I think I'll make the changes for the heroes list and have someone write some test mails.

Removing the LISTNAME from the Subject and not appending our friendly footer seems to have done the trick, DKIM signatures from e.g. suse.com can now be validated. OTOH, on the heroes list, even just mentioning the removal of LISTNAME from the Subject header caused a couple of people to complain. Not necessarily valid complaints, just an indication of what to expect.
Christian's suggestion is a good one, let us see how mailman3 handles it.

Making the change in mlmmj is easy, just remove LISTNAME/control/prefix and LISTNAME/control/footer, but I suggest we announce it first, maybe with a direct mail to each list.

Actions #13

Updated by mkamprianis over 3 years ago

pjessen wrote:

To test it, I think I'll make the changes for the heroes list and have someone write some test mails.

Removing the LISTNAME from the Subject and not appending our friendly footer seems to have done the trick, DKIM signatures from e.g. suse.com can now be validated. OTOH, on the heroes list, even just mentioning the removal of LISTNAME from the Subject header caused a couple of people to complain. Not necessarily valid complaints, just an indication of what to expect.
Christian's suggestion is a good one, let us see how mailman3 handles it.

Making the change in mlmmj is easy, just remove LISTNAME/control/prefix and LISTNAME/control/footer, but I suggest we announce it first, maybe with a direct mail to each list.

Thanks
I understand the complexity and the turbulence that any change brings. I have requested lowering the security output of (our) DKIM implementation to facilitate; but this will probably take time to be implemented - if it ever does, and will actually move the burden of signing in a specific way to the senders with unpredictable results.

I will subscribe to the lists and try, can I please have a mail with headers from bugzilla_noreply@suse.com? It should not be unsigned, so we need to identify which systems this mail passes through.

Actions #14

Updated by pjessen over 3 years ago

mkamprianis wrote:

I will subscribe to the lists and try, can I please have a mail with headers from bugzilla_noreply@suse.com? It should not be unsigned, so we need to identify which systems this mail passes through.

This is from an arbitrary mail, retrieved from the list archives. Let me know if it doesn't works this way, otherwise I can attach it as an .eml file.

From opensuse-bugs+bounces-907950-archive=lists4-intern.suse.de@opensuse.org  Thu Oct  1 00:14:44 2020
Return-Path: <opensuse-bugs+bounces-907950-archive=lists4-intern.suse.de@opensuse.org>
X-Original-To: archive@lists4-intern.suse.de
Delivered-To: archive@lists4-intern.suse.de
Received: from baloo.infra.opensuse.org (localhost [127.0.0.1])
        by lists5.opensuse.org (Postfix) with ESMTP id C5F1C11572;
        Thu,  1 Oct 2020 00:14:44 +0000 (UTC)
X-Original-To: opensuse-bugs@lists5.opensuse.org
Delivered-To: opensuse-bugs@lists5.opensuse.org
Received: from relay2.suse.de (unknown [195.135.221.27])
        by lists5.opensuse.org (Postfix) with ESMTP id 9B65811572
        for <opensuse-bugs@lists5.opensuse.org>; Thu,  1 Oct 2020 00:14:44 +0000 (UTC)
Received: from relay2.suse.de (localhost [127.0.0.1])
        by relay2.suse.de (Postfix) with ESMTP id C6CC42C253
        for <opensuse-bugs@lists5.opensuse.org>; Thu,  1 Oct 2020 00:14:48 +0000 (UTC)
X-Virus-Scanned: by amavisd-new at relay2.suse.de
X-Spam-Flag: NO
X-Spam-Score: -0.999
X-Spam-Level:
X-Spam-Status: No, score=-0.999 tagged_above=-9999 required=5
        tests=[ALL_TRUSTED=-1, HTML_MESSAGE=0.001]
        autolearn=no autolearn_force=no
Received: from relay2.suse.de ([127.0.0.1])
        by relay2.suse.de (relay2.suse.de [127.0.0.1]) (amavisd-new, port 10026)
        with ESMTP id u1o7Prul81vH for <opensuse-bugs@lists5.opensuse.org>;
        Thu,  1 Oct 2020 00:14:43 +0000 (UTC)
Received: from mx2.suse.de (unknown [149.44.161.68])
        (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
        (No client certificate requested)
        by relay2.suse.de (Postfix) with ESMTPS id 9C3E12C41D
        for <opensuse-bugs@lists5.opensuse.org>; Thu,  1 Oct 2020 00:12:23 +0000 (UTC)
Received: from novprvlin1091-suse.bugzilla-priv.suse.de (unknown [195.135.220.25])
        by mx2.suse.de (Postfix) with ESMTP id 0920CB020
        for <opensuse-bugs@opensuse.org>; Thu,  1 Oct 2020 00:12:19 +0000 (UTC)
From: bugzilla_noreply@suse.com
To: opensuse-bugs@opensuse.org
Subject: [Bug 1175289] azure-cli initial "help" behavior oddity
Date: Thu, 01 Oct 2020 00:12:18 +0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Classification: openSUSE
X-Bugzilla-Product: openSUSE Tumbleweed
X-Bugzilla-Component: Cloud:Tools
X-Bugzilla-Version: Current
X-Bugzilla-Keywords: 
X-Bugzilla-Severity: Minor
X-Bugzilla-Who: adrian.glaubitz@suse.com
X-Bugzilla-Status: IN_PROGRESS
X-Bugzilla-Priority: P5 - None
X-Bugzilla-Assigned-To: adrian.glaubitz@suse.com
X-Bugzilla-QA-Contact: qa-bugs@suse.de
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: 
X-Bugzilla-NTS-Support-Num: 
Message-ID: <bug-1175289-21960-lxROAQloYH@http.bugzilla.suse.com/>
In-Reply-To: <bug-1175289-21960@http.bugzilla.suse.com/>
References: <bug-1175289-21960@http.bugzilla.suse.com/>
Content-Type: multipart/alternative; boundary="1601511138.B5A5210.30991";
 charset="UTF-8"
X-Bugzilla-URL: http://bugzilla.suse.com/
Auto-Submitted: auto-generated
Precedence: bulk
Mailing-List: contact opensuse-bugs+help@opensuse.org; run by mlmmj
X-Mailinglist: opensuse-bugs
List-Post: <mailto:opensuse-bugs@opensuse.org>
List-Help: <mailto:opensuse-bugs+help@opensuse.org>
List-Subscribe:  <mailto:opensuse-bugs+subscribe@opensuse.org>
List-Unsubscribe:  <mailto:opensuse-bugs+unsubscribe@opensuse.org>
List-Owner: <mailto:opensuse-bugs+owner@opensuse.org>
List-Archive: <http://lists.opensuse.org/opensuse-bugs/>
X-MIME-Notice: attachments may have been removed from this message
MIME-Version: 1.0
Actions #15

Updated by mkamprianis over 3 years ago

Thanks
how are we doing with the rest of the lists?
We 're continuously attacked and we need to enable DMARC as soon as possible. Do you think that modifying the mailing lists is something you can do?

Thank you
Michalis

Actions #16

Updated by hellcp over 3 years ago

mkamprianis wrote:

Thanks
how are we doing with the rest of the lists?
We 're continuously attacked and we need to enable DMARC as soon as possible. Do you think that modifying the mailing lists is something you can do?

We discussed it during the last meeting and are planning switching over to mailman3 soon, I was thinking that maybe this weekend I could migrate everything if heroes list migration goes well in the coming days, and I am hoping oSLO slows down the mailing lists traffic somewhat, since the community will be busy doing something else. But today is only Monday, so who knows how well that ends up going ;)

Actions #17

Updated by pjessen over 3 years ago

I have written an email to announce the changes to everyone, it only needs to be sent:

From: "openSUSE mailing list admin" <ml-admin@opensuse.org>
To: ##EMAIL##
Subject:  [announcement] mailing list changes per ##DATE##
MIME-Version: 1.0
Content-Type:  text/plain; charset="iso-8859-1"

Dear subscriber of ##LIST##

we are planning a couple of more or less significant changes to the way we run
the openSUSE mailing lists:

* stop prepending the [LISTNAME] prefix to the Subject header
* stop appending a footer (typically with unsubscribe and owner information) to
  every posting.

The above will take effect as of ##WHEN##

By ceasing to make these changes to the original mail, it will be possible for
a receiving mailserver to verify DKIM signatures added by the original poster.
Currently, our changes to the email content prevent this verification which in
turn might cause mailservers to outright reject openSUSE mailing lists, when/if
instructed to do so by the sender's DMARC policy.

Overall, we don't expect this to have any noticable impact for you, unless you
have local filtering based on the [LISTNAME] prefix in the Subject header.  In
this case, we suggest you adapt your filters to use the "X-Mailing-List" instead.
While you are at it, it might be a good idea to also look for the "List-Id"
header, as we expect to start using that header when we switch the mailing list
manager to mailman3.

Any and all comments or questions are most welcome, but it is probably best to
direct them to the mailing list admin address:

mailto:ml-admin@opensuse.org


Best regards
Per Jessen
Member, openSUSE Heroes, mailing list admin

It might be sensible to decouple the change-sets and just do this one independently of the mailman3 migration.

Actions #18

Updated by Pharaoh_Atem over 3 years ago

I would really like to keep appending the footer in the mailing lists.

Could we have mailman just activate DMARC mitigations conditionally? That seems to be supported: https://mailman.readthedocs.io/en/release-3.1/src/mailman/handlers/docs/dmarc-mitigations.html

Also, it seems that the behavior of stripping DKIM headers by mailman is something that Fedora's instance does too: https://pagure.io/fedora-infra/ansible/blob/master/f/roles/mailman/templates/mailman.cfg.j2

Actions #19

Updated by mkamprianis over 3 years ago

Pharaoh_Atem wrote:

I would really like to keep appending the footer in the mailing lists.

Could we have mailman just activate DMARC mitigations conditionally? That seems to be supported: https://mailman.readthedocs.io/en/release-3.1/src/mailman/handlers/docs/dmarc-mitigations.html

I have no objection, but I also have no saying as I don't use the mailing list :)
Changing the From is THE suggested way to deal with the problem, but most mailing lists members don't like it.

Also, it seems that the behavior of stripping DKIM headers by mailman is something that Fedora's instance does too: https://pagure.io/fedora-infra/ansible/blob/master/f/roles/mailman/templates/mailman.cfg.j2

DMARC works like that:
-> is a DKIM signature valid in the message? If yes, accept, otherwise, do something (quarantine | reject)

If you strip the DKIM signature and you keep the same "From", you are in the "otherwise" territory; i.e. no valid DKIM signature in the message.

Actions #20

Updated by GeraldPfeifer over 3 years ago

Pharaoh_Atem wrote:

I would really like to keep appending the footer in the mailing lists.

Yes, though having to choose between that or munging the From: header I am very much in favor of maintaining the From: instead of appending a footer.

Rewriting the From: header is meddling too much IMHO.

What Per proposes here appears to be the only way to avoid messages getting blocked/lost when DMARC is activated (for suse.com or other domains, this is not specific to one domain).

Actions #21

Updated by mkamprianis over 3 years ago

Hi all
where are we with that? Do you think you can provide a target date?

Actions #22

Updated by Pharaoh_Atem over 3 years ago

GeraldPfeifer wrote:

Pharaoh_Atem wrote:

I would really like to keep appending the footer in the mailing lists.

Yes, though having to choose between that or munging the From: header I am very much in favor of maintaining the From: instead of appending a footer.

Rewriting the From: header is meddling too much IMHO.

What Per proposes here appears to be the only way to avoid messages getting blocked/lost when DMARC is activated (for suse.com or other domains, this is not specific to one domain).

I'm aware that this will affect other domains. However, note that with Mailman 3, people will be able to post from the HyperKitty interface to the mailing list if they are subscribed, and in that case the headers are going to be written as such:

From: Person via list <list@lists.opensuse.org>
Reply-To: Person <person@example.org>

My problem with not mangling the header and not appending the footer is that we can't ensure basic information is conveyed to everyone on the list in a convenient fashion that makes it difficult to ignore (e.g. appropriate conduct guidelines, unsubscription info, etc.)

In all Fedora lists, for example, we append the following to the message body:

_______________________________________________
<list> mailing list -- <list>@lists.fedoraproject.org
To unsubscribe send an email to <list>-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/<list>@lists.fedoraproject.org

And I would rather have something similar for all openSUSE lists.

Actions #23

Updated by pjessen over 3 years ago

Pharaoh_Atem wrote:

My problem with not mangling the header and not appending the footer is that we can't ensure basic information is conveyed to everyone on the list in a convenient fashion that makes it difficult to ignore (e.g. appropriate conduct guidelines, unsubscription info, etc.)

In all Fedora lists, for example, we append the following to the message body:
[snip]
And I would rather have something similar for all openSUSE lists.

I agree wholeheartedly. Michalis, is there any way we can accommodate this with the new DMARC/DKIM setup?

Actions #24

Updated by mkamprianis over 3 years ago

pjessen wrote:

Pharaoh_Atem wrote:

My problem with not mangling the header and not appending the footer is that we can't ensure basic information is conveyed to everyone on the list in a convenient fashion that makes it difficult to ignore (e.g. appropriate conduct guidelines, unsubscription info, etc.)

In all Fedora lists, for example, we append the following to the message body:
[snip]
And I would rather have something similar for all openSUSE lists.

I agree wholeheartedly. Michalis, is there any way we can accommodate this with the new DMARC/DKIM setup?

Hi
No, this is not in our plans due to technical limitations.

Update on our side, we go live with DMARC implementation on Friday the 6th of November. The policy will be to quarantine any mail that fails DKIM signature.

Actions #25

Updated by Pharaoh_Atem over 3 years ago

mkamprianis wrote:

pjessen wrote:

Pharaoh_Atem wrote:

My problem with not mangling the header and not appending the footer is that we can't ensure basic information is conveyed to everyone on the list in a convenient fashion that makes it difficult to ignore (e.g. appropriate conduct guidelines, unsubscription info, etc.)

In all Fedora lists, for example, we append the following to the message body:
[snip]
And I would rather have something similar for all openSUSE lists.

I agree wholeheartedly. Michalis, is there any way we can accommodate this with the new DMARC/DKIM setup?

Hi
No, this is not in our plans due to technical limitations.

Update on our side, we go live with DMARC implementation on Friday the 6th of November. The policy will be to quarantine any mail that fails DKIM signature.

Then we'll just have to use DMARC mitigations for the openSUSE mailing lists. Our transition to Mailman 3 will give us the option of turning those on selectively (that is, for those who set reject/quarantine for their DMARC policy).

Actions #26

Updated by mkamprianis over 3 years ago

All
our plan is shifted a bit; we are going live on Tuesday the 11th of November.
I urge you to speed up the migration to mailman3 and implementation of the DMARC mitigations, or to remove the footers / subject modifications. We know this works, as is obvious in major mailing lists such as kernel.org.

Can you share an update on how you plan to move forward?

Actions #27

Updated by pjessen over 3 years ago

  • Status changed from Feedback to In Progress

FYI, after today's change of the "suse.com" DMARC policy to quarantine, on our mx[12].o.o mails from bugzilla are now being indicated as 'quarantine' as they do not have a DKIM signature. (see https://progress.opensuse.org/issues/72133#note-10).

Actions #28

Updated by mdaltin over 3 years ago

Hi everyone!

My 2 cents here:

DMARC record was changed on 12th of November to implement the quarantine policy for DMARC
mx?.suse.de two days later: 14th of November

Maybe this: https://bugs.launchpad.net/mailman/+bug/557493 might be of interest too, regarding the apparent DKIM signature stripping by mailman.

Ciao, M.

Actions #29

Updated by pjessen over 3 years ago

mdaltin wrote:

DMARC record was changed on 12th of November to implement the quarantine policy for DMARC
mx?.suse.de two days later: 14th of November

Maybe this: https://bugs.launchpad.net/mailman/+bug/557493 might be of interest too, regarding the apparent DKIM signature stripping by mailman.

I have just now changed mailman to not remove the DKIM header. Stasiek said he set it up as a test, but forgot about it :-)
I would normally advise against such changes on a Friday evening, but in this case it might be useful - less traffic over the weekend etc.

Actions #30

Updated by GeraldPfeifer over 3 years ago

Great, looks like we are converging. :-)

One observation: When I send mail to project@lists.opensuse.org from my gp@suse.com
address, the mail I get back from the list contains the standard footer.

Doesn't this break DKIM (and hence DMARC)?

https://lists.opensuse.org/archives/list/project@lists.opensuse.org/message/SVTA5DKD4F4ZQDJWAR3T6VUDQ4IGRLNV/
is one such example, though the "cool" new mailing archive does not show that.

Actions #31

Updated by pjessen over 3 years ago

GeraldPfeifer wrote:

Great, looks like we are converging. :-)

One observation: When I send mail to project@lists.opensuse.org from my gp@suse.com
address, the mail I get back from the list contains the standard footer.

Doesn't this break DKIM (and hence DMARC)?

Yup it does. As does the prepending on the LISTNAME in the Subject header.

The latter is "easily" changed, whereas I have been unable to override the footer.

Essentially - leaving the DKIM header intact does not change anything.

Actions #32

Updated by GeraldPfeifer over 3 years ago

  • Subject changed from RE: need to contact opensuse.org mail admins to openSUSE mailing lists break domains like suse.com that use DMARC/DKIM

pjessen wrote:

Yup it does. As does the prepending on the LISTNAME in the Subject header.

The latter is easily changed, whereas I have been unable to override the footer.

Since we need to address both (the Subject rewriting and footer addition), can you go ahead and change the former for now? I believe that was announced as part of the migration to mailman3, so should not be much surprise to regulars at least.

As for the footer, does mailman3 feature specific handling of domains with DMARC/DKIM, or is this a global setting? If the latter, do we have any choice but not adding those footers?

Full disclosure: I am personally quite vested here since right now mails of mine (that I send as gp@suse.com) are prone to rejection/quarantine already, depending on how members' and contributors' mail servers are configured. So I appreciate any help.

Actions #33

Updated by pjessen over 3 years ago

GeraldPfeifer wrote:

pjessen wrote:

Yup it does. As does the prepending on the LISTNAME in the Subject header.

The latter is easily changed, whereas I have been unable to override the footer.

Since we need to address both (the Subject rewriting and footer addition), can you go ahead and change the former for now? I believe that was announced as part of the migration to mail

The truth is - changing the Subject means visiting each individual list page, and we have 100+ of them :-)

It won't improve on the DMARC/DKIM situation either. Some have already been updated though.

I have asked on the mailman3 users list, and the per-list override of the footer IS supposed to work. I am testing it on bugs.lists.o.o

Actions #34

Updated by pjessen over 3 years ago

Have removed subject prefix up until cloud.lists.o.o
Have removed subject prefix up until go.lists.o.o
Have removed subject prefix up until kubic-bugs.lists.o.o
Have removed subject prefix up until news.lists.o.o
Have removed subject prefix up until programming.lists.o.o

Actions #35

Updated by pjessen over 3 years ago

Have removed subject prefix up until selinux.lists.o.o

Actions #36

Updated by GeraldPfeifer over 3 years ago

pjessen wrote:

Have removed subject prefix up until selinux.lists.o.o

Thank you, Per!

Actions #37

Updated by pjessen over 3 years ago

GeraldPfeifer wrote:

pjessen wrote:

Have removed subject prefix up until selinux.lists.o.o

Thank you, Per!

:-) we'll get there, a little bit at a time.

Now up to 'updates'.
Now up to 'wiki'.

Actions #38

Updated by pjessen over 3 years ago

pjessen wrote:

GeraldPfeifer wrote:

pjessen wrote:

Have removed subject prefix up until selinux.lists.o.o

Thank you, Per!

:-) we'll get there, a little bit at a time.

Now up to 'updates'.
Now up to 'wiki'.

Okay, LISTNAME subject prefix removed from all of the lists. Still no solution wrt the footers.

Actions #39

Updated by pjessen over 3 years ago

  • Status changed from In Progress to Feedback

pjessen wrote:

Okay, LISTNAME subject prefix removed from all of the lists. Still no solution wrt the footers.

Umm, I was just looking at the DKIM headers from suse.com - I am a little surprised to see "Subject" not being included in the headers being hashed:

From bugzilla:
h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:
mime-version:mime-version:content-type:content-type;

From an individual suse.com address:
h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:
mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding

Is this intentional?

Actions #40

Updated by mkamprianis over 3 years ago

pjessen wrote:

pjessen wrote:

Okay, LISTNAME subject prefix removed from all of the lists. Still no solution wrt the footers.

Umm, I was just looking at the DKIM headers from suse.com - I am a little surprised to see "Subject" not being included in the headers being hashed:

From bugzilla:
h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:
mime-version:mime-version:content-type:content-type;

From an individual suse.com address:
h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:
mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding

Is this intentional?

Hi
this is not intentional, I did not know and this is not the right way forward. Moreno will chase it internally to be fixed.

Actions #41

Updated by bmwiedemann over 3 years ago

"Subject" not being included in the headers being hashed

This was a test via
https://gitlab.suse.de/OPS-Service/salt-dmz/-/merge_requests/201

Stop including subject in DKIM signature to make it easier to pass MLs that modify Subject with [ML-name]

We can revert that, if it causes problems.

Actions #42

Updated by GeraldPfeifer over 3 years ago

I want to make sure everybody is on the same page, so here how the situation looks
from my perspective as a user:

  • Subject headers on opensuse.org lists are no longer modified.
  • Mail bodies are still modified:

Both mails I sent to @lists.opensuse.org from gp@suse.com made it back to gp@suse.com
which is subscribed to those lists (with the footer added per the above).

Does this mean (a) DMARC has not yet been enabled for suse.com or (b) modifying the body
actually is not an issue or (c) something else?

Actions #43

Updated by pjessen over 3 years ago

GeraldPfeifer wrote:

I want to make sure everybody is on the same page, so here how the situation looks
from my perspective as a user:

  • Subject headers on opensuse.org lists are no longer modified.
  • Mail bodies are still modified:

Confirmed. That is the current situation.

Both mails I sent to @lists.opensuse.org from gp@suse.com made it back to gp@suse.com
which is subscribed to those lists (with the footer added per the above).

In principle, they should probably both have been quarantined. SUSE inbound mails are handled by mimecast.com.

Does this mean (a) DMARC has not yet been enabled for suse.com or (b) modifying the body
actually is not an issue or (c) something else?

a) the DMARC policy at suse.com was published sometime mid-November, policy is 'quarantine'.
Can be seen with a simple dns lookup - "dig _dmarc.suse.com txt"
b) modifying the body remains an issue
Can be seen in the log, e.g. when Google quarantines a mail, or when other mails are rejected.
c) for DMARC to work, the receiving mailserver has to implement it too.

Actions #44

Updated by pjessen over 3 years ago

  • Status changed from Feedback to In Progress

a) the DMARC policy at suse.com was published sometime mid-November, policy is 'quarantine'.
Can be seen with a simple dns lookup - "dig _dmarc.suse.com txt"
b) modifying the body remains an issue
Can be seen in the log, e.g. when Google quarantines a mail, or when other mails are rejected.
c) for DMARC to work, the receiving mailserver has to implement it too.

Yesterday I updated our footer template for all lists, it is now empty which should mean no DMARC/DKIM rejections.

Actions #45

Updated by pjessen about 3 years ago

  • Status changed from In Progress to Resolved

It's been a little more than a month and I have not received or seen a single complaint.

Actions #46

Updated by pjessen about 3 years ago

  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF