openSUSE admin

2020-02-04 #opensuse-admin - heroes meeting

[20:00:06] <klein> hello there

[20:00:24] <kbabioch> hi there @ all

[20:00:35] <jdsn> hi

[20:00:39] <tuanpembual> hi

[20:00:46] <cboltz> hi, and welcome to the heroes meeting

[20:01:01] <cboltz> the (planned) topics are on https://progress.opensuse.org/issues/61862

[20:01:45] <cboltz> let's start with the Q&A - does someone from the community have a question?

[20:03:21] <cboltz> doesn't look so ;-)

[20:03:31] <cboltz> next topic then:

[20:03:34] <cboltz> status reports

[20:03:42] <cboltz> I'll start with a small one:

[20:04:02] <cboltz> I finally tested updating the wiki server to 15.1 yesterday (on my test VM)

[20:04:20] <cboltz> besides a (easily fixed) dependency problem, everything worked as expected

[20:04:38] <cboltz> which also means I'll update the public wiki server soon (probably tomorrow evening)

[20:05:02] <kl_eisbaer> great!

[20:05:19] <jdsn> I have an announcement about widehat.o.o: https://cloud.nanuna.net/s/F87esNEKEBdDzHx

[20:05:29] <jdsn> so the machine is there and I to performance testing and move it to the datacenter after the repos have been preseeded via our internal network (faster)

[20:05:44] <jdsn> s/I to/I do/

[20:07:22] <kl_eisbaer> 2nd time great!

[20:07:54] * kbabioch applauds ;-)

[20:07:57] <cboltz> indeed, that should solve the storage shortage for a while :-)

[20:08:38] <jdsn> you found the hidden size hint

[20:09:33] <lcp> hi

[20:10:02] <Eighth_Doctor> Hey!

[20:11:02] <kl_eisbaer> Other status reports?

[20:11:02] <cboltz> lcp: I think you also can report something ;-)

[20:12:16] <cboltz> spoiler alert for part of lcp's work: http://jekyll.infra.opensuse.org/ (needs VPN)

[20:12:19] <lcp> news-o-o and planet-o-o work :tm: since today, and I will have to ask for test subdomains for them

[20:13:02] <kbabioch> (also has a report / question about certificates)

[20:14:17] <lcp> matrix.i.o.o has a working salt config, just needs a little bit newer packages to actually work

[20:17:01] <cboltz> sounds good :-)

[20:17:25] <kbabioch> i'm wondering whether anyone is actively looking into the certificate situation ... we got some warnings ~ 2 weeks ago. I "fixed" crtmgr, but certs have not been re-deployed everywhere. we still get some warnings about expiring certificates (in 21 days) from musafa and proxy-prv1 for instance ...

[20:17:49] <tampakrap> hello! I came for the meeting, can someone send me the backlog please?

[20:18:27] <cboltz> yes, let me paste it...

[20:19:01] <kl_eisbaer> kbabioch: mufasa and proxy-prv1 are the same machine. How do you deploy the certs from crtmgr? Via SSH or via Salt?

[20:19:01] <cboltz> https://paste.opensuse.org/a7de8fe7

[20:19:06] <tampakrap> mufasa and proxy-prv1 afaik are the same machine, and afaik there is no automation to update the cert there because there was some issue that I don't remember (but I may be wrong

[20:19:07] <Pharaoh_Atem> I just got access to the VPN, so now I'm trying to get started with infra stuff

[20:19:21] <darix> kl_eisbaer: ssh

[20:19:37] <Pharaoh_Atem> the first thing I'm planning to do is get Ipsilon working and connected to our IPA server

[20:19:39] <kl_eisbaer> so just have a look if the deployment fails...

[20:20:38] <Pharaoh_Atem> also, I'd like to start figuring out how to upgrade our FreeIPA deployment

[20:20:52] <Pharaoh_Atem> but I don't know anything about how it was set up and how it is managed

[20:21:11] <darix> Pharaoh_Atem: it is a POC manual install of a fedora

[20:21:20] <lcp> darix: I tested getting links in SRs, they don't seem to appear in any case (even with osc sr --update-links)

[20:21:24] <Pharaoh_Atem> darix: do we have backups of the system?

[20:21:34] <darix> lcp: you just dont SR ... you just create links

[20:21:51] <lcp> hm, alright

[20:22:05] <klein> Pharaoh_Atem: I would say that we should have a new machine, to install everything fresh, so we do not break anything

[20:22:36] <lcp> salt managed fedora vm? >:D

[20:22:39] <kl_eisbaer> Pharaoh_Atem: who needs backups, if you just need restore? ;-)

[20:23:03] <mstroeder> Well, at opensuse heroes meeting we had a breakout session also regarding user management. This also affects SSH key distribution. We want to discuss that now in a structured fashion?

[20:23:29] <Pharaoh_Atem> klein: we can do that if we want

[20:23:42] <Pharaoh_Atem> probably a good opportunity to use some of the Ansible roles officially available for it and plug that into Salt

[20:23:43] <darix> this is kinda unsorted

[20:23:50] <darix> this is like 4-5 topics in parallel

[20:24:44] <mstroeder> Since mid of 2018 there's an Æ-DIR PoC installation running on Leap 15.1. Preferrably it would be used with my own NSS/PAM demon aehostd which would also allow to hunk out sssd.

[20:25:03] <Pharaoh_Atem> "hunk out"?

[20:25:16] <mstroeder> BTW: A customer partially funded Python 3 migration. So Æ-DIR runs without Python2.

[20:25:24] <Pharaoh_Atem> nice, congrats

[20:25:28] <kl_eisbaer> mstroeder: is there a merge request for it in our salt repo ?

[20:25:36] <mstroeder> Yes, replace sssd with aehostd: https://www.ae-dir.com/aehostd.html

[20:25:49] <Pharaoh_Atem> mstroeder: Æ-DIR looks nice, but I wish it wasn't backed by OpenLDAP

[20:25:58] <Pharaoh_Atem> that community is awful :(

[20:26:14] <mstroeder> It is backed by OpenLDAP and I deal with upstream developers. No problem.

[20:26:25] <Pharaoh_Atem> though it could be argued that I just have PTSD over dealing with hyc :(

[20:26:40] <mstroeder> Well, I'm part of the OpenLDAP community. Do you consider me personally to be awful?

[20:26:58] <Pharaoh_Atem> nah, you're good people :)

[20:27:20] <Pharaoh_Atem> honestly, I don't care too terribly much whether we do FreeIPA or Æ-DIR

[20:27:31] <Pharaoh_Atem> I have experience with FreeIPA, so I'm not much help with the latter

[20:27:37] <Pharaoh_Atem> but Ipsilon can work with either

[20:28:09] <mstroeder> The Æ-DIR installation is ready for thorough testing. It needs nicer server certs, so the browser does not prompt people.

[20:28:13] <klein> I think something that runs on openSUSE should be the best coice

[20:28:17] <klein> *choice

[20:28:27] * Pharaoh_Atem shrugs

[20:28:35] <Pharaoh_Atem> if we wanted to keep FreeIPA, I'd help port it

[20:28:38] <mstroeder> Current Æ-DIR PoC installation is a single provider and a single consumer. But just give me more VMs and it's HA.

[20:29:02] <mstroeder> Æ-DIR installation is fully automated using ansible: https://gitlab.infra.opensuse.org/stroeder/infra-ae-dir

[20:29:16] <Pharaoh_Atem> I've gotten very good at handling porting back and forth between Fedora and openSUSE now

[20:29:21] <Pharaoh_Atem> and I'm doing a bunch of work in Fedora and openSUSE to reduce the differences

[20:29:42] <mstroeder> Can we vote on that?

[20:30:01] * Pharaoh_Atem personally would like to see Æ-DIR in Fedora repos as well as openSUSE ones :)

[20:30:21] <mstroeder> Another suggestion for replacing SSH authorized keys: https://ekca.stroeder.com/

[20:30:31] <klein> cboltz: you own the meeting, vote that? Please drive the topics, as Darix pointed we were discussing more than one thing, and for this FreeIPA/Aedir we need some end

[20:30:49] <mstroeder> Running Æ-DIR on CentOS 7.7 is fully supported.

[20:30:50] <Pharaoh_Atem> mstroeder: my personal requirement would be that the solution we go to has a path to being in Factory

[20:31:17] <Pharaoh_Atem> whether cboltz or anyone else agrees is a different matter, but I like stuff we use being in Factory eventually :)

[20:31:40] <cboltz> given how long we have that "test setup" of Æ-Dir, it's probably really a good idea to vote on it instead of "testing" it for another year ;-)

[20:31:46] <lcp> well, rawdog wasn't in factory, we are getting rid of that >:D

[20:31:47] <mstroeder> Installation is ansible-based. Software packaging varies depending on the platform. For openSUSE/SLE there are native packages: https://build.opensuse.org/project/show/home:stroeder:AE-DIR

[20:32:10] <cboltz> the only problem is that AFAIK not too many people have looked at Æ-Dir yet, wich might make voting hard

[20:32:32] <mstroeder> As said: The PoC installation already runs and is maintained by me since 08/2018.

[20:32:38] <cboltz> so let me first ask: is someone here who wants/needs to look at it before voting?

[20:32:55] <Pharaoh_Atem> mstroeder: if we go with Æ-DIR, would you contribute a module for Ipsilon to have easy setup like the one for IPA?

[20:33:01] <klein> maybe mstroeder can do a presentation for us?

[20:33:06] <darix> was it even on the agenda for today?

[20:33:34] <lcp> can we have a heroes conference where we get a talk about aedir and freeipa from both sides?

[20:33:50] <Pharaoh_Atem> lcp: we should get ab to come if we do that :)

[20:34:11] <darix> and are you aware that freeipa does more than just users for us?

[20:34:12] <mstroeder> I did look at Ipsilon in 2018 because a Python-based IdP is very attractive for me. But personally I consider upstream Ipsilon to be dead.

[20:34:50] <Pharaoh_Atem> mstroeder: it's not dead, the developers are working on rebuilding FAS at the moment, and Ipsilon's big development last year was moving to Python 3

[20:35:11] <darix> ok I am out for today.

[20:35:32] <Pharaoh_Atem> Securitas is being worked on to replace Fedora's FAS2 right now, because that urgently needs replacing

[20:35:40] <Pharaoh_Atem> RHEL 6 + turbogears makes for a sad situation :(

[20:35:41] <klein> ok, people... darix has a fair point. The agenda really don't have this discussion about aedir x freeipa

[20:35:59] <mstroeder> Yes, I know that FreeIPA also does DNS for infra.opensuse.org. At the heroes meeting we also discussed refactoring DNS based on PowerDNS.

[20:36:00] <klein> please check it out: https://progress.opensuse.org/issues/61862

[20:36:30] <klein> I think we shoudl follow the agenda

[20:36:32] <Pharaoh_Atem> for the ssh key management, we should use sssd or aehostd

[20:36:54] <cboltz> klein: right, that's one of our typical "extended status report" which we wanted to get rid off ;-)

[20:36:56] <Pharaoh_Atem> salt based provisioning is a bad way to do it, because it's not as simple to self-service

[20:37:17] <mstroeder> The agenda has SSH key management in salt with authorized keys for root. The heroes team has to decide whether that's enough. If yes, you can hunk out any user management from the equation.

[20:37:42] <Pharaoh_Atem> I think both mstroeder and I would argue that key management should be part of user management

[20:37:59] <Pharaoh_Atem> I work at a place that doesn't do it that way, and it makes life pretty miserable at times

[20:38:47] <mstroeder> Other opinions?

[20:38:52] <Pharaoh_Atem> we use something like this at work: https://github.com/fuhry/openssh-ldap-authkeys

[20:38:58] <Pharaoh_Atem> it's... not as much fun as you'd think

[20:40:25] <cboltz> salting the authorized_keys deployment shouldn't be too bad, but I also tend to think that having the keys in FreeIPA / Æ-Dir is better

[20:40:37] <cboltz> (ignoring the sssd "fun" for now)

[20:40:42] <kbabioch> i don't have strong opinion and what we end up using. i have two points to make, though: a.) ssh key management via salt doesn't work well ... many old keys lying around, which no one removes, only new keys are added all of the time ... b.) we have a very active member caring a lot about identity management (mstroeder), so would argue to take him up on his offer and let him deal with this ;-)

[20:41:08] <cboltz> however, I wonder if it makes sense to keep our $whatever-admin groups for sudo permissions

[20:41:28] <cboltz> or if it would be easier to give sudo permissions to individual users listed in the pillar/id/ files

[20:41:32] <kbabioch> managing authorized_keys with configuraiton management doesn't work well in my experience ... maybe if it is *really* automated and runs regularly ...

[20:41:47] <kbabioch> usually people will only trigger an update when they want to add some key(s) ... old keys are never removed in this way

[20:41:55] <Pharaoh_Atem> it didn't work for us at $DAYJOB, which is why it still comes from LDAP for us

[20:42:23] <kbabioch> i also didn't / doesn't work for my previous $DAYJOB ;-)

[20:42:50] <Pharaoh_Atem> we use Puppet, which is strictly worse than Salt, but I think the same problems would occur ;)

[20:43:13] <cboltz> sounds like experience decided on this part ;-)

[20:43:50] <cboltz> what about the $whatever-admins group vs. individual users ("responsible" in pillar/id/*)?

[20:43:51] <Pharaoh_Atem> we don't necessarily *have* to user logins map to it (as openssh-ldap-authkeys indicates), but personally, I think it's a bad idea to allow M:1 relationship

[20:44:14] <Pharaoh_Atem> but if everyone else is okay with it, we can just use that and map LDAP groups to root

[20:44:31] <Pharaoh_Atem> that'll work with FreeIPA or Æ-DIR

[20:44:35] <Pharaoh_Atem> since it's just LDAP we're speaking

[20:44:43] <lcp> sounds better than salting responsible users to root tbh

[20:44:45] <mstroeder> Æ-DIR/aehostd also takes care of sudoers stuff. But aehostd even provides a virtual group ae-vgrp-role-setup if you don't want to deal with sudo at all: https://www.ae-dir.com/aehostd.html#maps

[20:47:23] <cboltz> the advantage of using "responsible" from pillar/id/* would be that we could auto-generate the sudo config, and don't have to maintain a group somewhere else

[20:48:02] <klein> keep this settings "split" (in more than one tool/control) could not be a good idea

[20:48:44] * kl_eisbaer has only 10 min left - is there anything I can help with ?

[20:49:23] <malcolmlewis> kl_eisbaer, Forum migration and SSO

[20:49:29] <cboltz> well, provide "the answer[tm]" ;-) - I have a feeling that this topic won't be solved today

[20:49:56] <kl_eisbaer> malcolmlewis: I have to admit: I can not really help with both

[20:50:13] <kl_eisbaer> SSO (aka SAML) is something you want to ask darix

[20:50:22] <kl_eisbaer> forum migration is per's topic

[20:50:28] <malcolmlewis> kl_eisbaer, ahh ok

[20:50:53] <kl_eisbaer> but I bet that we will not get a working DB dump this year

[20:51:28] <kl_eisbaer> my recommendation would therefor be to install something new and provide the old forums in a read-only mode for some time

[20:51:32] <malcolmlewis> kl_eisbaer, plan B new forum

[20:51:44] <kl_eisbaer> malcolmlewis: yes, exactly

[20:51:50] <cboltz> kl_eisbaer: I hope that you are wrong with your bet, but given previous experience, well... ;-)

[20:52:13] <malcolmlewis> that was my expectation a long time ago....

[20:52:29] <mstroeder> Since forums seem to be a never-ending story: How about setting the old forum to read-only and start with something new (e.g. discourse like darix suggested)?

[20:52:37] <kl_eisbaer> cboltz: just keep in mind that the same instance is powering the Novell/SUSE/Microfocus forums. MF-IT would have to sort out the data from the dump before they can provide it....

[20:53:14] <malcolmlewis> kl_eisbaer, SUSE have an extension out to June/July for their forum

[20:53:55] <lcp> is working with mf on migration this bad?

[20:54:00] <lcp> jeez

[20:54:17] <klein> nohhh... its worst :-)

[20:54:59] <malcolmlewis> I think I can get blood out of a stone quicker....

[20:55:12] <cboltz> I can tell some stories about the wiki migration some years ago - and what I've seen for the forums migration is indeed much worse :-(

[20:55:22] <kl_eisbaer> I can partly understand them, if the DB dump contains a lot of data which is not openSUSE related. In this case they either need to find a way to sort stuff out automatically (which, I guess, is not possible) or do it "by hand"

[20:56:12] <kl_eisbaer> and in the last case, just think about how many entries someone needs to scan.

[20:56:26] <lcp> thankfully lizards is only really used by yast team

[20:56:27] <malcolmlewis> kl_eisbaer, over 40K

[20:56:39] <lcp> which uses a blog that uses jekyll so we can host it

[20:56:54] <kl_eisbaer> So - while I do not have any single contact or insight - I guess this is the main reason. But they will never tell you....

[20:57:19] <malcolmlewis> but no user data in vB, so if that goes, nobody can sign into the forums

[20:58:05] <mstroeder> Are the user accounts also stored in MF's eDirectory?

[20:58:11] <cboltz> right, obviously we'll need to have SSO again

[20:58:17] <malcolmlewis> we just habe username/email and that's tied to SAML

[20:58:31] <malcolmlewis> yup

[20:58:37] <kl_eisbaer> v8 is like our redmine or other tools: it does not really hold user data. Someone needs to authenticate against LDAP or eDirectory - and once this is done, the user is created automatically.

[20:58:37] <malcolmlewis> cboltz, ^^

[20:58:59] <cboltz> mstroeder: yes, but we have a copy of that in NBG (aka login2.o.o)

[20:59:28] <malcolmlewis> OK, I will see what the mods/admins think about new forum and read only vB

[20:59:54] <mstroeder> WebSSO is only stateless front-end stuff. The main point is that you would need the dump from eDirectory including the clear-text passwords (or pre-hashed for the new target platform).

[21:00:03] <lcp> I can salt discourse if yo need that

[21:00:13] <Pharaoh_Atem> mstroeder: btw, in case you didn't know, I did package ipsilon for openSUSE almost immediately after it was ported to Python 3

[21:00:47] <cboltz> mstroeder: AFAIK there's a copy of the LDAP content in Nuremberg (which login2.o.o and the buildservice use)

[21:00:57] <kl_eisbaer> ok - time for me. CU!

[21:01:30] <mstroeder> And the "LDAP content" is constantly synchronized?

[21:01:53] <cboltz> I don't know the details, but AFAIK - yes

[21:02:12] <klein> I am pretty sure that we only have ldap proxies

[21:02:23] <klein> so, there is some data, but, as a "cache" only

[21:02:23] <lcp> that explains why there is a delay between registering and ability to login to bs

[21:02:26] <malcolmlewis> mstroeder, cboltz I guess this is why all the mf/hpe users arrive on the forum

[21:03:11] <klein> we have a project to migrade the LDAP used by bugzilla to an internal solution (started by kbabioch )

[21:05:45] <cboltz> without knowing any details on this - are there any plans to make this "internal solution" available for openSUSE services?

[21:06:03] <cboltz> ("available" as in "can be used to authentificate users")

[21:06:19] <Pharaoh_Atem> heh

[21:06:54] <lcp> or are we on our own with our own auth setup? >:D

[21:07:38] <lcp> post mf era infra has a chance of not being an absolute nightmare tbh

[21:08:10] <lcp> it can also be even more of a nightmare, if we try hard enough

[21:08:37] <cboltz> I fully agree (on the first one), but the split could easily qualify as a nightmare

[21:08:39] <mstroeder> AFAICS the "internal solution" is for bugzilla etc. because also SLE customers are using it.

[21:08:56] <mstroeder> An openSUSE authc solution would be limited to openSUSE community users.

[21:09:07] <Pharaoh_Atem> that's a good thing IMO

[21:09:14] <lcp> it wouldn't be a bad way to do things

[21:09:16] <kbabioch> mstroeder: good guess, but it is even more complicated than that ;-)

[21:09:17] <Pharaoh_Atem> this weird tie-in with corporate stuff has been horrible for us

[21:09:20] <lcp> we just would need bugzilla sso

[21:09:35] <lcp> somebody ask rh to share that with SUSE, eh?

[21:09:41] <kbabioch> openSUSE should definitely try / aim for as much independence as possible ... just to be safe from any such things that are currently going on ;-)

[21:09:51] <Pharaoh_Atem> somebody at SUSE should talk to someone at RH about that

[21:09:56] <klein> I think openSUSE should be separated...

[21:10:03] <Pharaoh_Atem> I'm pretty bloody certain that RH would share the mods they made

[21:10:12] <cboltz> well, bugzilla.suse.com == bugzilla.opensuse.org - which also means we'll probably have to share the auth stuff

[21:10:38] <lcp> we can sso from 2 auth providers on bugzilla

[21:10:39] <Pharaoh_Atem> cboltz: the way RHBZ does it is that it can accept SSO from either Red Hat or Fedora and link to the RHBZ account

[21:10:40] <lcp> if we wanted to split

[21:10:41] <mstroeder> What's so special about bugzilla SSO? You could add a SSO frontend handler in front of bugzilla and let bugzilla just look at REMOTE_USER var.

[21:11:57] <Pharaoh_Atem> because bugzilla's REMOTE_USER support sucks

[21:11:59] <cboltz> I wouldn't be too surprised if bugzilla already looks at HTTP_X_USERNAME

[21:12:08] <Pharaoh_Atem> built-in, it basically supports only CAS for SSO

[21:12:19] <kbabioch> we shouldn't get lost in technical details until we know which auth backend / protocol / api we end up using ;-)

[21:12:22] <Pharaoh_Atem> Red Hat and Mozilla independently added other auth support to their stuff

[21:13:26] <lcp> I wouldn't mind if we had ipsylon and securitas set up for the purpose for now, just so we have some actual fallback

[21:13:41] <lcp> then we can think about if we want to take this further with any migration

[21:13:42] <Pharaoh_Atem> I can look into doing that

[21:14:13] <cboltz> as kbabioch said - let's wait until we know what we _need_ ;-)

[21:14:16] <Pharaoh_Atem> I'll have to chat with the Fedora AAA team about what we need to do to make Securitas nicer for SUSE things

[21:14:22] <Pharaoh_Atem> err openSUSE things :)

[21:17:32] <cboltz> so - is there anything we can actually do for the forums move? Or should we switch to the next topic (mail setup)?

[21:18:02] <malcolmlewis> cboltz, I guess a test setup is needed?

[21:18:31] <klein> mail setup doesn't depend on kl_eisbaer

[21:18:33] <klein> ?

[21:18:47] <malcolmlewis> for the forum software

[21:18:47] <cboltz> Per already has a VM, and already got the forum php files - but without the database dump, that doesn't really help

[21:19:05] <malcolmlewis> cboltz, move to a new platform

[21:19:44] <malcolmlewis> when get data (which will need to be done again) put a read-only vB instance up

[21:20:20] <cboltz> I'd prefer to avoid such a split because it makes things quite messy

[21:20:28] <cboltz> (you know old-en.o.o?)

[21:20:41] <cboltz> the current plan is

[21:20:51] <malcolmlewis> cboltz, then I guess a waiting game

[21:21:01] <cboltz> - finally get the database dump (yeah, that's a waiting game :-( )

[21:21:28] <lcp> can't old-en-o-o just be moved to github nowadays? it's only useful for historical purposes of checking the history of the project

[21:21:31] <cboltz> - setup vB with minimal changes (goal: quick, not introducing errors)

[21:21:38] <lcp> it's not very useful as a wiki

[21:21:40] <cboltz> - then migrate to something better

[21:22:15] <cboltz> lcp: in theory yes, but in practise keeping it with mediawiki is easier ;-)

[21:22:35] <lcp> true

[21:23:14] <malcolmlewis> cboltz, that's the easy part, it's the login part....

[21:23:16] <lcp> although the site gets progressively more broken the more guo plays with the theme

[21:24:47] <cboltz> malcolmlewis: I know, but I hope that we get that sorted out "easily" when we have the database dump

[21:24:58] <robin_listas> lcp: for users, it is easier to read old-en... on wiki, even if read only.

[21:24:59] <cboltz> I mean, it can't be harder than getting the database dump ;-))

[21:25:46] <malcolmlewis> cboltz, true, ok, will followup with emails to per etc, you need to move on to new topics ;)

[21:26:45] <cboltz> yes, we have the email setup on the agenda

[21:27:04] <cboltz> originally that was meant to discuss anna/elsa (our current outgoing relays)

[21:27:47] <cboltz> but I've seen that we now have mx[12].infra.o.o

[21:28:20] <cboltz> (at least those VMs appeared in salt recently ;-)

[21:29:04] <cboltz> is someone here who knows any details about them?

[21:29:23] <jdsn> they are an idea by kl_eisbaer

[21:29:33] <jdsn> and I also talked to him about it

[21:30:06] <mstroeder> I guess mx[12].infra.o.o are the result of the mail breakout session at heroes meeting: https://progress.opensuse.org/issues/59923

[21:30:13] <jdsn> part of the the idea is, that the new widehat is powerful enough to host one of the mx'es and maybe some more vms $outside

[21:31:12] <jdsn> this could also solve the issue, that some of the current IPs are not trusted by telekom

[21:31:21] <jdsn> which was the original topic :)

[21:32:00] <cboltz> are these VMs meant for incoming or outgoing mails?

[21:32:19] <jdsn> to be defined

[21:32:45] <jdsn> at the moment its just ideas we had in the hallway

[21:33:12] <cboltz> ah, ok

[21:33:41] <jdsn> if it makes sense we could do both

[21:34:00] <cboltz> IMHO ideally we should have one set of VMs for incoming mails, and another one for outgoing

[21:34:11] <mstroeder> mx[12].infra.o.o are meant as inbound MX for opensuse.org to get independent of SUSE infrastructure.

[21:34:19] <jdsn> agreed

[21:36:28] <mstroeder> On outbound MTAs I still would recommend to be super-picky with forward/reverse DNS, outgoing IP address and HELO name. You never know what receiving MXs might refuse the other day.

[21:37:19] <jdsn> indeed

[21:38:05] <jdsn> I will talk to kl_eisbaer about that - we both run our own mailservers, so we can combine our experience

[21:38:36] <jdsn> I mean about also deploying a set of relays

[21:39:19] <cboltz> currently outgoing mails are sent by anna/elsa. haproxy also runs there - and that already means that changing the IP of static.o.o breaks mail delivery

[21:39:45] <mstroeder> IP reputation can easily become a problem. How about monitoring the usual blacklists for own outgoing IP addresses?

[21:39:46] <cboltz> so moving outgoing mails to a separate IP (and separate VM) would make sense

[21:40:27] <jdsn> +1 to both

[21:41:15] <mstroeder> Yes, always use dedicated outgoing IP addresses. Do not use that for anything else, keep IP addresses constant. Try to get them white-listed.

[21:41:56] <cboltz> we should also monitor the number of rejected outgoing mails - blacklists are not the only thing that can break mail delivery

[21:42:45] <jdsn> cboltz: so you want to become the recipient of postmaster@ ?

[21:44:06] <cboltz> I didn't say that ;-)

[21:44:18] <mstroeder> Is postmaster@opensuse.org a mail distribution list or a functional mailbox?

[21:44:33] <cboltz> postmaster@ should probably go to progress.o.o like admin@ does

[21:44:34] <jdsn> "rejected outgoing mails" either return to sender or to postmaster in the end

[21:45:02] <cboltz> I was more thinking about monitoring the log

[21:45:22] <cboltz> the rejected mails themself will probably go to lists.o.o in > 90% of all cases

[21:45:33] <jdsn> cboltz: thats what I meant - these mails will return anyway

[21:45:59] <jdsn> so monitoring them is just creating duplicated noise

[21:46:16] <cboltz> maybe I should explain it in another way:

[21:46:29] <jdsn> or put differently - we should not drop such mails, they should end up on someones table

[21:47:13] <cboltz> count number of rejects/bounces per hour - if that number surprisingly increases a lot, we should get an alert

[21:47:42] <jdsn> ok got it - and agree :)

[21:49:42] <cboltz> is that enough input for you and Lars?

[21:49:55] <jdsn> yes I guess so

[21:50:13] * jdsn has to leave in a few minutes

[21:50:27] <cboltz> ok, then let's end the meeting ;-)

[21:50:27] <jdsn> are there any other thoughts for that topic?

[21:50:34] <jdsn> :)

[21:53:03] <jdsn> ok - thanks guys, good n8

[21:53:27] <mstroeder> Bye.

[21:53:36] <cboltz> good night, and thanks everybody for joining the meeting!