2017-12-05-heroes-meeting.txt - openSUSE admin - openSUSE Project Management Tool

communication #27500 » 2017-12-05-heroes-meeting.txt

IRC log of the meeting - cboltz, 2017-12-05 21:57

 -12-05 #opensuse-admin - Heroes meeting
 :03:41  <tampakrap> meeting time!
 :04:02  <pjessen> hello
 :04:09  <kl_eisbaer> katnip: whatever is useful and provides links to a) a Mailer b) a Browser and c) a Console ;-)
 :04:19  <kl_eisbaer> huhu
 :04:26  <katnip> gotcha :)
 :04:27  <pjessen> all KDE here.
 :04:32  <katnip> same
 :05:34  <tampakrap> so let's start?
 :05:36  <tampakrap> cboltz: around?
 :05:42  <cboltz> yes
 :05:56  <mmaher_home> hi
 :06:06  <katnip> hi
 :06:41  <tampakrap> cboltz: wanna moderate?
 :06:55  <cboltz> no problem ;-)
 :07:18  <cboltz> let's start with questions from the community
 :07:23  <cboltz> does someone have a question?
 :09:13  <tampakrap> apparently not
 :09:22  <cboltz> doesn't look so, so let's continue with
 :09:25  <cboltz> using deprecated service as default haproxy answer (like https://crashdb.opensuse.org/) ?
 :10:05  <kl_eisbaer> this is my current idea to "announce" to users that a service is no longer available
 :10:26  <kl_eisbaer> something we might use for whatever webservice we discontinue next :-)
 :10:36  <tampakrap> this is the default backend?
 :10:45  <kl_eisbaer> tampakrap: no, not the default one
 :10:59  <tampakrap> good, +1 from me then
 :11:01  <cboltz> IMHO it's much better and more informative than a plain 404 page :-) so +1 from me
 :11:12  <kl_eisbaer> My current idea is to show this page just for services we had in the past but no longer (like crashdb or apparmor)
 :11:14  <tampakrap> how long to keep it?
 :11:27  <kl_eisbaer> tampakrap: good question
 :11:32  <tampakrap> 6 moths is fine?
 :11:32  <kl_eisbaer> IMHO a year should be enough
 :11:40  <kl_eisbaer> but that includes that we track it....
 :11:49  <kl_eisbaer> ...and afterward remove it from the DNS and from haprox
 :11:50  <kl_eisbaer> y
 :12:17  <kl_eisbaer> Just one note: the page get's delivered via haproxy directly - so we don't need any special service for it
 :12:25  <cboltz> I'd argue that subdomains and that page are cheap - just keep it forever ;-)
 :12:41  <kl_eisbaer> there is one little problem, tough: there is a size limit for it
 :13:25  <kl_eisbaer> I am very, very close with the page to the size limit (so even one or two more words might be too much)
 :14:00  <kl_eisbaer> in case we get over the size limit, haproxy would not show anything (and also does not log anything, which was very "interesting to debug" in the beginning)
 :14:12  <kl_eisbaer> so I hope the explanation there is enough
 :14:28  <tampakrap> yep it's perfect
 :14:55  <kl_eisbaer> ok, so "one step closer to the 'deprecation of services' SLA" ;-)
 :15:13  <tampakrap> yep
 :15:24  <kl_eisbaer> if anyone has a better idea/layout, feel free to contact me
 :15:33  <kl_eisbaer> other than that: that's it for the topic at the moment
 :15:58  <cboltz> there's one small issue - there's a small grey artefact below the geeko
 :16:10  <cboltz> kl_eisbaer: I'll send you a fixed image later
 :16:33  <kl_eisbaer> cboltz: just send me an update
 :16:41  <kl_eisbaer> merci
 :17:04  <cboltz> ok, so let's continue with the next topic: how to proceed with redmine, connect and icc?
 :17:27  <kl_eisbaer> ...and conference ... ;-)
 :17:36  <kl_eisbaer> ...and svn ...
 :17:38  <kl_eisbaer> ...and ...
 :17:40  <tampakrap> I added icc, can we move it to the group of crashdb and gallery please?
 :17:52  <tampakrap> conference is wip, new machine is dale.i.o.o already given to the admins
 :18:02  <kl_eisbaer> oh, fine :-)
 :18:16  <tampakrap> I'd say 3 months (that is before the next osc)
 :18:35  <kl_eisbaer> no objections from my side against icc
 :19:02  <tampakrap> svn is in the suse-dmz right?
 :19:02  <kl_eisbaer> IMHO we should think about a meeting once a year to discuss the services we want to take down
 :19:15  <kl_eisbaer> tampakrap: at the moment, yes. I contacted jdsn already and provided svn2 for him
 :19:29  <kl_eisbaer> he just did not find any time yet to work on it
 :19:39  <tampakrap> I lost you, to work on what?
 :19:55  <tampakrap> svn2 is the hostname of the suse-dmz machine that provides svn.o.o and kernel.o.o/kernel.suse.com
 :19:57  <kl_eisbaer> jsdn, the current maintainer of svn, needs to work on svn2 - the new machine
 :20:21  <kl_eisbaer> oh, sorry. Than it was the other way around and the new one is svn while the old one is svn2 ;-)
 :20:45  <tampakrap> ah yes now I remember we talked about this again
 :20:53  <tampakrap> so the new svn machine is sle12 right?
 :20:57  <tampakrap> or leap?
 :21:04  <kl_eisbaer> leap 42.3  - as always
 :21:16  <tampakrap> okay, so why drop it then?
 :21:19  <kl_eisbaer> maybe we can get rid of svn.opensuse.org at all
 :21:23  <tampakrap> ahhh
 :21:27  <tampakrap> now I get you
 :21:44  <tampakrap> so I could send a mail to -factory and -project if anybody is still using it
 :21:56  <tampakrap> if not, it is going away in let's say 3 months, sounds good?
 :22:02  <tampakrap> I could write a blog post as well
 :22:17  <pjessen> maybe check the access logs?
 :22:21  <kl_eisbaer> tampakrap: you can do it maybe easier in the first run: just check the public repos and their last committer
 :22:27  <cboltz> it seems some of the SVN repos are still used
 :22:35  <cboltz> opensuse-doc had the last commit 8 weeks ago
 :22:57  <kl_eisbaer> jip - but maybe we can convince Karl to use github :-)
 :23:22  <cboltz> opensuse-i18n also had a commit 2 months ago - funnily also by Karl ;-)
 :23:32  <tampakrap> and apparently it is the only one
 :23:46  <tampakrap> translations are migrated to git already
 :23:54  <kl_eisbaer> so maybe talking with him is enough - for the public repos
 :24:04  <kl_eisbaer> I did not check if there are any hidden repos, yet
 :24:41  <tampakrap> okay I'll do it
 :24:47  <kl_eisbaer> jip: there are some
 :24:54  <kl_eisbaer> but all of them are very, very old...
 :25:05  <kl_eisbaer> thanks tampakrap
 :26:07  <kl_eisbaer> One question, still: should we schedule a meeting during OSC (?) where we decide about discontinued services ?
 :26:48  <tampakrap> yes please
 :27:36  <tampakrap> redmine is also on my list to update to sle12, it will take time though, so any help will be welcome
 :28:00  <tampakrap> I am working on the new voting system that the board requested with high priority these days
 :28:06  <kl_eisbaer> tampakrap: what about next week ?
 :28:47  <tampakrap> sure, assuming I finish the voting system first
 :29:00  <kl_eisbaer> tampakrap: no problem, I have enough other tasks  ;-)
 :29:12  <kl_eisbaer> tampakrap: just ping me, if you need help
 :29:47  <tampakrap> https://progress.opensuse.org/issues/28902 filed this for svn.o.o
 :30:28  <kl_eisbaer> ok, thanks. I guess that is it for the deprecated or "to be moved" services ?
 :30:38  <tampakrap> connect.o.o I assume the agreement is still to drop it as soon as we have the new voting system I am working on with scarabeus, and as soon as we have the new mailing system from heinlein right?
 :31:08  <kl_eisbaer> I don't know, sorry
 :31:17  <kl_eisbaer> I just had the old task in progress to handle connect.o.o
 :31:30  <kl_eisbaer> so I started to work on a elgg package and reserved a machine for it
 :31:32  <tampakrap> looking at the salt master, the only remaining sle11 machines are the narwals, which could go to salt directly and replace them all with leap
 :31:47  <kl_eisbaer> boosters ?
 :31:54  <tampakrap> boosters is connect
 :31:55  <kl_eisbaer> community?
 :32:02  <tampakrap> and also community that I don't know what is doing tbh
 :32:03  <kl_eisbaer> pontifex ;-)
 :32:46  <kl_eisbaer> community is running some vhosts like counter.o.o or the IRC bot
 :32:58  <kl_eisbaer> it's a mix of accounts and services
 :33:37  <tampakrap> ah and osc-collab (not in salt), the maintainers also have been given a leap machine to move it almost a year ago
 :33:48  <tampakrap> so I suppose it is time to send them a mail and put a deadline on them
 :33:58  <kl_eisbaer> tampakrap: jip, totally agreed
 :34:24  <tampakrap> we really can't maintain sle11 any more, especially when 15 is knocking our door
 :34:29  <kl_eisbaer> tampakrap: just one question: did you contact any of the maintainers of events.o.o already ?
 :34:38  <tampakrap> yes, henne
 :34:43  <kl_eisbaer> ah, ok.
 :34:59  <kl_eisbaer> JFYI: this machine is already broken
 :35:07  <tampakrap> I know
 :35:13  <kl_eisbaer> ok
 :35:37  <kl_eisbaer> I guess we can finish this topic, than ?
 :35:41  <tampakrap> yes
 :36:08  <kl_eisbaer> "  Priorisation of TODOs " can IMHO be skipped, too.
 :36:08  <cboltz> ok, so next topic - Priorisation of TODOs
 :36:33  <cboltz> ok, then let's continue with
 :36:35  <kl_eisbaer> was my topic before I filled the others :-)
 :36:39  <cboltz> use of https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Machines and the pages behind
 :36:53  <kl_eisbaer> Jip - more or less something FYI only
 :37:06  <cboltz> have a look at http://paste.opensuse.org/372359e3
 :37:28  <cboltz> that's how I'd put the "standardized" parts into salt (pillar/id/)
 :37:30  <kl_eisbaer> cboltz: nice :-)
 :38:02  <cboltz> this would mean that we can get rid of most pages you created ;-)
 :38:03  <kl_eisbaer> So we might extend the machine starting page with a link to that general Saltyfied list
 :38:07  <tampakrap> kl_eisbaer: /root/bin/generate_redmine_wiki.sh can you commit that script to the salt repo as well please? we have the bin/ directory with such tools
 :38:11  <kl_eisbaer> cboltz: nope, not really
 :38:18  <cboltz> the content is still useful, just moved to salt
 :38:19  <kl_eisbaer> tampakrap: of course
 :38:50  <kl_eisbaer> cboltz: I see the wiki pages more for "problem solvers" and people, who work on a machine only if Salt is broken ;-)
 :39:12  <tampakrap> wiki page is always useful to have, yes
 :39:15  <kl_eisbaer> cboltz: ...or how would you implement something like for  https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Monitorinfraopensuseorg
 :39:16  <cboltz> I'd argue that everybody should have a checkout of the salt repo at hand ;-)
 :39:19  <kl_eisbaer> for example ?
 :39:53  <kl_eisbaer> The monitoring already links to the special wiki pages for each machine, btw.
 :40:01  <tampakrap> cboltz: my suse-it colleagues might need to work on one of those machines and they might not have vpn or salt handy unfurtunately
 :40:22  <tampakrap> so having the info in a place that doesn't require vpn is always nice
 :40:29  <cboltz> we could solve that by having a checkout of the salt repo on a webserver
 :40:41  <cboltz> but I don't want to have the information duplicated at salt and wiki
 :40:49  <kl_eisbaer> so - while I really appreciate to have most informations in Salt, sometimes it's just handy to give admins some old, ugly text files at hand to start with their job
 :41:04  <tampakrap> as long as it is generated/automated, I don't mind having it duplicate
 :41:08  <pjessen> +1
 :41:17  <pjessen> to bother
 :41:19  <pjessen> both
 :41:30  <kl_eisbaer> pjessen: +1 for cboltz or +1 for tampakrap ?
 :41:47  <pjessen> you and tampa
 :41:50  <cboltz> kl_eisbaer: for the additonal information (like on the Monitorinfraopensuseorg page), we can (and should) still use wiki pages, but
 :42:13  <cboltz> - those pages could have a more readable name (for example just "Monitor" or "Monitoring")
 :42:27  <cboltz> - several machines can link to the same page
 :42:42  <kl_eisbaer> cboltz: the problem is, that I want to have a documentation link for every machine that is monitored
 :43:09  <kl_eisbaer> if there is a more general information provided, someone could always create additional pages and link to them
 :43:13  <kl_eisbaer> see https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Olafinfraopensuseorg as examle
 :43:45  <kl_eisbaer> but using Salt to collect some basic information of the machine is very cool and helpful
 :44:21  <kl_eisbaer> just think about :
 :44:29  <kl_eisbaer> Admin notices a problem with a machine on https://monitor.opensuse.org/icinga/
 :44:52  <kl_eisbaer> Now he just needs to click on the "Extra host notes" (the folder icon beside the host name)
 :45:22  <kl_eisbaer> and he get's redirected to the wiki page of the machine - where he hopefully finds all useful information on how to debug/fix or getting help
 :45:58  <pjessen> sounds good
 :46:01  <cboltz> nice integration ;-)
 :46:13  <cboltz> but - we could also get this by serving pillar/id/$machine ;-)
 :46:15  <kl_eisbaer> most of the current wiki pages of the machines just contain "useless" information that could obviosly also provided by salt, yes
 :46:29  <cboltz> (or a formatted version of those files)
 :47:04  <kl_eisbaer> but I hope that the administrators of the machine understand that people, who want to fix their wiki server will have other problems than to search in Salt for the information ;-)
 :47:06  <tampakrap> kl_eisbaer: https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Sarabiinfraopensuseorg (as an example) this page is wrong, I edit it via the script or I can edit hte wiki page directly?
 :47:33  <kl_eisbaer> cboltz: but if you don't want to use it, feel free to ignore it. I know that I sometimes are very happy to have such information at hand
 :47:48  <kl_eisbaer> tampakrap: the machine pages should be edited by hand
 :48:03  <tampakrap> cool
 :48:06  <kl_eisbaer> just the overview page is created by the script - as I did not want to miss a machine that is monitored
 :48:16  <tampakrap> perfect
 :48:49  <kl_eisbaer> ...and the monitoring is still not correct, as we miss most of the machines in Provo
 :48:54  <kl_eisbaer> but that's another topic
 :49:39  <kl_eisbaer> For it was just important to point out that there is now a direct connection between a monitored machine and some documentation of the machine
 :50:07  <kl_eisbaer> if the specific wiki page is used or not is something I like to leaf up to the admin of the machine
 :50:09  <heroes-bot> RECOVERY: MySQL WSREP recv on galera1.infra.opensuse.org - OK wsrep_local_recv_queue_avg = 0.195652 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
 :50:25  <kl_eisbaer> ^ juhu, backup job done ;-) (sorry)
 :50:28  <tampakrap> yep got it
 :50:32  <cboltz> one problem we have with this way that we split information across multiple places
 :50:51  <kl_eisbaer> cboltz: you might be right - but I would leave that up to the maintainer of the machine
 :51:01  <tampakrap> I would prefer to have the info on ldap instead of salt tbh, and populate specific info on salt
 :51:10  <kl_eisbaer> cboltz: or you provide me a similar way to easily create/update and provide the documentation in Salt
 :51:10  <tampakrap> instead of keeping salt as the central point of such data
 :51:52  <kl_eisbaer> I already tried to find out how much information we could store in Freeipa - but there is IMHO not really much possible
 :52:28  <kl_eisbaer> location, arch, OS-Version, MAC address, host group and certificates are ...... Information that could also be stored in Salt. Here is where I agree with cboltz
 :53:02  <kl_eisbaer> but that does not tell me why (for example) a memcached is running on the machine or what other service is using it and how
 :53:21  <tampakrap> I agree as well with that, I am talking about data like cnames and responsible people
 :53:35  <kl_eisbaer> tampakrap: ah, yes.
 :53:40  <tampakrap> they could be in ldap, populated automatically in salt (and even better in salt-mine)
 :53:58  <kl_eisbaer> tampakrap: https://freeipa.infra.opensuse.org/ipa/ui/#/e/host/search
 :54:01  <tampakrap> then we can generate monitoring data and wiki pages from salt-mine
 :54:10  <kl_eisbaer> => I already started to use it ;-)
 :54:15  <cboltz> if you agree with the pillar/id/ additions in http://paste.opensuse.org/372359e3 _and_ agree that we should _move_ (not duplicate) the information to salt, I'm probably able to come up with a script that creates nice HTML pages for each machine you could link from the monitoring
 :54:21  <kl_eisbaer> for example, the host groups
 :54:44  <cboltz> the "documentation:" part can still link to the wiki, but those links should be role-specific, not machine-specific IMHO
 :54:45  <kl_eisbaer> I created "apache_servers, nginx_servers" ....and combined them in ... "web-servers"
 :55:30  <kl_eisbaer> So now, if you click on the web-servers host group and check the "Indirect Membership", you get all our web servers, regardless if they are running nginx or apache
 :55:52  <tampakrap> so the host groups can be populated as roles in salt
 :55:57  <kl_eisbaer> ...and btw: if you add a new host in the host group, freeipa automatically created the DNS entries for you
 :56:04  <kl_eisbaer> tampakrap: jip, exactly
 :56:12  <tampakrap> perfect, I like
 :56:47  <tampakrap> hopefully freeipa can create the sudoers rules so we can get rid of them from salt
 :56:53  <kl_eisbaer> ...and other services like the monitoring (for service and hostgroups) or salt could use this information from ldap
 :57:10  <kl_eisbaer> tampakrap: https://freeipa.infra.opensuse.org/ipa/ui/#/e/sudorule/search
 :58:02  <kl_eisbaer> so:yes - you can create sudoers via freeipa - you just need to think about groups and the normal organizational stuff
 :58:28  <tampakrap> got it
 :59:05  <kl_eisbaer> btw: I already created a "machine" user group that has a lower requirement on password changes
 :59:26  <kl_eisbaer> so users in that group don't need to change their password too often
 :00:03  * cboltz wonders if he's the only one who doesn't want to spread things over several places (yes, that's a vote against using freeipa for things we could do in salt ;-)
 :00:03  <kl_eisbaer> but I guess we can skip the freeipa tutorial to another meeting ;-)
 :00:23  <kl_eisbaer> cboltz: you know KISS ?
 :00:36  <tampakrap> I filed also https://gitlab.infra.opensuse.org/infra/salt/merge_requests/127 and https://gitlab.infra.opensuse.org/infra/salt/merge_requests/128 to generate the monitoring contacts from the ldap users and from the salt roles
 :00:37  <pjessen> my favourite principle
 :00:39  <kl_eisbaer> take the best tool for a job - and let the other tools make use of it
 :01:19  <kl_eisbaer> cboltz: using Salt for everything is like going into your wine garden with a truck ...
 :01:25  <cboltz> I know KISS - but IMHO keeping everything at one place is part of it (unless the additional tool is _really_ worth it)
 :01:53  <kl_eisbaer> cboltz: redundance is sometimes needed - but what you should keep in mind the the authority
 :02:38  <kl_eisbaer> there is one authoritative service - the other might have duplicated data, but they need to keep in sync with the authority
 :03:21  <kl_eisbaer> at the moment, neither Salt, nor Freeipa nor any other service knows everything about the network, the services, their users and admins
 :03:44  <kl_eisbaer> but if you wisely combine all of them, you get a very, very good picture
 :04:08  <heroes-bot> PROBLEM: MySQL WSREP recv on galera1.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 1.875776 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
 :04:15  <kl_eisbaer> in the end, it's just a question how you present that information to the user in the end
 :05:24  <kl_eisbaer> ...and for me, this sometimes means: "clicky bunty wins" ;-)
 :07:05  <kl_eisbaer> more questions/remarks?
 :07:16  <kl_eisbaer> I don't think we come to a conclusion here today
 :07:28  <cboltz> I get your point, but would still prefer to have things at one place unless there are very good reasons for a separate place
 :07:43  <kl_eisbaer> cboltz: agreed
 :07:48  <cboltz> I could even create HTML pages for each machine from pillar/id/* if we decide to use that way ;-)
 :08:01  <cboltz> so you'd have something you can link from the monitoring
 :08:23  <cboltz> and those pages could include links to additional wiki pages with more information
 :08:29  <kl_eisbaer> at the moment, my reason is that I want important information about a host at a single page - available via a link from the monitoring. And this information should be easily editable and extensible
 :08:32  <tampakrap> cboltz: I'd suggest to invest some time on salt-mine instead of pillar/id/*
 :09:05  <kl_eisbaer> cboltz: fine with me - I just did what my time permits
 :09:18  <kl_eisbaer> cboltz: if you have better solutions, feel free - I will not object
 :09:35  <kl_eisbaer> cboltz: at the moment, it's just me using these pages anyway
 :09:40  <tampakrap> then the machines will know also data about each other, eg monitoring will know all the IPs of all the machines, the web servers will know the hostnames of their database servers, the haproxy servers will know the IPs of their webservers etc
 :10:25  <kl_eisbaer> cboltz: I'm just building "my infrastructure" as easy as possible - for me.
 :10:35  <cboltz> I have a feeling that we have different opinions what is a "better solution" ;-)
 :10:43  <cboltz> there's nothing wrong with that
 :10:47  <kl_eisbaer> cboltz: feel free to convince me that your solution is as good and you got me ;-)
 :11:14  <cboltz> so before I spend time to move your wiki content to salt, I'd like to know that we'll go that way ;-)
 :11:29  <kl_eisbaer> cboltz: I just hope that "my" solution gives you a good idea what I want to achieve
 :11:39  <cboltz> (I'll of course provide one or two examples so that you can see what I'm going to do)
 :11:51  <kl_eisbaer> cboltz: yes, please
 :12:01  <pjessen> I'm looking forward to it, too
 :12:07  <cboltz> yes, you basically want
 :12:17  <kl_eisbaer> cboltz: at least you should now have an idea what I expect from it ?
 :12:31  <cboltz> yes
 :12:42  <kl_eisbaer> If I need to update a wiki page or a text file is not relevant for me
 :12:52  <kl_eisbaer> btw: can I include pictures in salt?
 :13:15  <kl_eisbaer> => just joking - I know ASCII art :-)
 :13:24  <tampakrap> base64
 :13:30  <kl_eisbaer> hm.
 :13:31  <cboltz> lol
 :13:49  <kl_eisbaer> I was thinking about some basic UML or schema graphics
 :14:02  <cboltz> seriously: good question for salt itsself (I'd say just "git add picture.jpg"), but for the rendered page adding an <img> tag is probably easy
 :14:11  <kl_eisbaer> like [ server1 ] ---speaks with ---> [server2]
 :14:48  <kl_eisbaer> jip, if we can collect - if needed - such pictures somewhere in git and just point to them, I guess this should work
 :15:37  <cboltz> my solution can (and probably will) still include links to the wiki or $whatever_makes_sense - but those wiki pages will be service-specific (think of a pagename like "monitoring") instead of machine-specific
 :15:56  <kl_eisbaer> ok - fine with me
 :16:47  <kl_eisbaer> Next topic ?
 :17:23  <cboltz> no objections ;-)
 :17:28  *** Ada_Lovelace has joined #opensuse-admin
 :17:41  <cboltz> next topic is the offsite meeting
 :17:50  <cboltz> hi Sarah!
 :17:56  <Ada_Lovelace> Hi
 :18:06  <tampakrap> are we having one before the conference or we are going straight for the conf?
 :18:16  <Ada_Lovelace> I have forgotten the time...
 :18:46  <cboltz> I'd prefer a meeting before the conference
 :19:04  <cboltz> at the conference, everybody is busy enough, so we won't get too many things done
 :19:05  <Ada_Lovelace> Last month we spoke about February.
 :19:42  <kl_eisbaer> jip, I agree
 :20:04  <tampakrap> feb is fine
 :20:07  <kl_eisbaer> this time I would love to see  some "work groups" - getting things done
 :20:30  <kl_eisbaer> ...and than we can present that outcome during the conference :-)
 :20:32  <cboltz> the funny thing is that my only free weekend in February is Feb 17/18 - the other weekends I'm already busy with various other stuff
 :20:49  <tampakrap> so we're going for nuremberg at feb?
 :21:07  <kl_eisbaer> tampakrap: what about Prague ?
 :21:23  <tampakrap> sure, prague it is!
 :21:26  <kl_eisbaer> any preferences, anyone ?
 :22:03  <cboltz> depends if we want to optimize travel or if we want to switch locations ;-)
 :22:07  <pjessen> I can't promise - 17/18 is right in the middle of Sportferien
 :22:39  <kl_eisbaer> cboltz: Insheim ?
 :23:12  <cboltz> I'm not sure if my parents would like that idea ;-)
 :23:18  <kl_eisbaer> pjessen: I guess you will have the longest travel of us - so what would be your preferred date and location ?
 :23:47  <tampakrap> otherwise we go for 3-4 march
 :24:21  <pjessen> it shouldn't depend on me, but 24/25 would be better. 3/4 March is also good
 :24:54  <kl_eisbaer> cboltz, Ada_Lovelace ? 3/4 March ?
 :25:09  <Ada_Lovelace> 3/4 March is ok for me
 :25:25  <cboltz> 3/4 March is also ok for me
 :25:55  <kl_eisbaer> nice!
 :25:56  <kl_eisbaer> location ?
 :26:38  <Ada_Lovelace> Per didn't join us before. He should say what he like (his home or Prague). ;-)
 :27:11  <pjessen> I have to decline arranging anything for ZH, too many things going. My personal favourite would be Nuernberg
 :27:25  <kl_eisbaer> ok, fine with me
 :27:36  <Ada_Lovelace> No long way for me...
 :27:50  <kl_eisbaer> If everyone agrees, I will clarify the location in Nuremberg for the 3/4 of March
 :28:03  <katnip> i'm a fish out of water here
 :28:07  <kl_eisbaer> ...and also ask for travel and lodging support
 :28:28  <tampakrap> I agree but I would say let's give pjessen some time to change his mind so we can have it in prague this time :P
 :28:28  <kl_eisbaer> katnip: want to join ? :-)
 :28:41  <katnip> im in the US :/
 :28:43  <kl_eisbaer> tampakrap: you are free to bring enough beer with you
 :28:57  <kl_eisbaer> katnip: that's an argument, not a problem ;-)
 :29:35  <katnip> kl_eisbaer: i seriously don't think it would work out :/
 :29:56  <kl_eisbaer> katnip: if you don't want to jump into a train or boat, we could easily start a video conf or chat for you
 :30:06  <kl_eisbaer> s/train/plane/
 :30:24  * cboltz already wondered when oversea trains were invented
 :30:28  <kl_eisbaer> now that we have some servers under our control in the US .....
 :30:32  <katnip> the chat would work or maybe video
 :30:37  <pjessen> eurotunnel?
 :30:44  <kl_eisbaer> cboltz: I'm just ahead of time :-)
 :30:57  <cboltz> pjessen: that's undersea ;-)
 :31:00  <kl_eisbaer> katnip: cool - putting you on the list :-)
 :31:12  <katnip> ok :)
 :32:00  <pjessen> ah, details.
 :32:31  <pjessen> oversea trains = railway bridges ?  oka\y, getting too silly
 :33:41  <kl_eisbaer> next topic ?
 :33:48  <cboltz> yes
 :33:54  <cboltz> status reports about everything
 :34:43  <tampakrap> I'll go first
 :34:48  <tampakrap> salt has tests!!!!!
 :34:57  * tampakrap claps loud
 :35:33  <tampakrap> https://gitlab.infra.opensuse.org/infra/salt/pipelines/585/builds
 :35:34  <cboltz> related: http://paste.opensuse.org/15812913 ;-)
 :36:34  <tampakrap> so, during hackweek I worked on validation / syntax checking, there are a bunch of hacks introduced since upstream doesn't have the proper tooling
 :36:50  <tampakrap> special thanks to cboltz for having the strong nerves to review my code
 :37:31  <cboltz> ... and to tampakrap for not killing me after some evil reviews ;-)
 :37:33  <tampakrap> we validate that pillar/id/*.sls has all the necessary data and that they are correct, we check that pillar/role/*.sls are actually assigned roles
 :37:55  <tampakrap> and an overall show_highstate which is a quick way to render the overall code
 :38:00  <tampakrap> and perform syntax checking
 :38:18  <tampakrap> so, we don't have actual functionality tests, but at least we have progress
 :38:42  <tampakrap> also we have a deployment job, that runs only for the production branch and tells the master to pull the new code
 :38:45  <tampakrap> that's it
 :39:07  <kl_eisbaer> wow, that's awesome! :D
 :39:24  <kl_eisbaer> tampakrap: can you please make a posting out of it?
 :39:25  <tampakrap> on a related note: https://gitlab.infra.opensuse.org/infra/salt/merge_requests I have open MRs for chrony and for generating the contact cfgs, feel free to review
 :39:34  <kl_eisbaer> that really looks like cool stuff
 :39:43  <tampakrap> kl_eisbaer: you're on my mind, I wanted to tell it to the team meeting first :)
 :39:51  <kl_eisbaer> :D
 :40:24  <cboltz> I'll continue with another salt topic ;-)
 :40:28  <cboltz> I worked on doing the monitoring setup (client side) with salt
 :40:30  <cboltz> so if you want to add a check to /etc/nrpe.d/ or need to add a package to the zypper whitelist, you can now do it by adding some pillar data in salt
 :41:09  <cboltz> while doing that, I also cleaned up the package whitelist a lot (thanks to Theo for helping with that)
 :41:10  <tampakrap> cboltz: I don't like that the data are in salt, can we move it to ldap please? HAHAHA okay I'll stop now
 :41:41  <cboltz> tampakrap: I'll happily review your merge request ;-)
 :43:25  <cboltz> as soon as someone answers my questions in https://gitlab.infra.opensuse.org/infra/salt/issues/3 and https://gitlab.infra.opensuse.org/infra/salt/issues/3 I'll also add the package list and the /etc/nrpe.d/ checks of all machines to salt
 :43:38  <cboltz> s/3/2/ in the first link
 :44:08  <heroes-bot> RECOVERY: MySQL WSREP recv on galera1.infra.opensuse.org - OK wsrep_local_recv_queue_avg = 0.000000 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
 :44:20  <tampakrap> okay but please move these tickets to the progress-admin, the gitlab issue tracker was supposed to redirect to progress (until upstream broke it)
 :44:30  <tampakrap> so I'll close the ticketing system there completely
 :45:24  <cboltz> I'd hope for a quick answer so that we can close them without needing to move them ;-)
 :46:20  <tampakrap> fair enough
 :46:25  <kl_eisbaer> cboltz: let's discuss this tomorrow
 :46:42  <cboltz> ok
 :46:52  <kl_eisbaer> the good thing is: the monitoring will instantly know if something breaks ;-)
 :47:45  <cboltz> salt won't remove any packages or /etc/nrpe.d/ checks, so in theory nothing can break ;-)
 :47:54  <cboltz> we'll see if practise disagrees
 :48:39  <kl_eisbaer> I've another status report .... https://progress.opensuse.org/news/63
 :49:04  <kl_eisbaer> I hope to finish the migration of pontifex - aka download.o.o - on thursday, finally
 :49:34  <pjessen> Guys, I have to go - no status to report, really.  I have reserved 3/4 March in my calendar.
 :49:54  <kl_eisbaer> at the moment, I'm still fighting with "knapsack" modules and the database (esp. repmgr), but I've still good hope to finish this until tomorrow night
 :50:03  <kl_eisbaer> pjessen: thank you for your time!
 :50:19  <kl_eisbaer> pjessen: any objections to become MX master ?
 :50:31  <pjessen> MX master?
 :50:44  <kl_eisbaer> pjessen: topic "own MX for openSUSE ? "
 :50:57  <pjessen> Sure, I'll be happy to look after that.
 :51:04  <kl_eisbaer> I guess it's time to isolate the master MX from SUSE-IT and bring it under our control
 :51:23  <cboltz> agreed, but...
 :51:36  <kl_eisbaer> it might be a forwarding of list traffic to baloo and the other stuff to Heinlein in the end
 :51:59  <pjessen> It's what I do professionally anyway, so ...
 :52:02  <kl_eisbaer> but this way we can get rid of the ping - pong with SUSE-IT regarding the redirects
 :52:15  <cboltz> I was thinking about the other way round, so that Heinlein {c,w}ould do the spam filtering
 :52:22  <kl_eisbaer> ...and I talked already with Christian, who has no objections.
 :52:34  <tampakrap> ah so we can get rid of the shuttle net from baloo right?
 :52:38  <kl_eisbaer> So the only thing that we need to clarify is a) if we want and b) if legal agrees
 :52:53  <kl_eisbaer> tampakrap: yes, right
 :52:59  <Ada_Lovelace> Heinlein is doing something...
 :53:18  <Ada_Lovelace> He has tested something with my opensuse.org alias...
 :53:18  <kl_eisbaer> cboltz: in this case we need to figure out how Heinlein forwards the other stuff back to us
 :53:26  <cboltz> mmaher_home: do you have any news from Heinlein?
 :54:09  <mmaher_home> cboltz: the technical person wants to contact me this week. thats the last status i have.
 :54:09  <kl_eisbaer> pjessen: but I don't want to stop you - let's move this discussion to the mailing list ?
 :54:45  <Ada_Lovelace> 2 weeks ago I received notifications by mailbox.org admin.
 :54:58  <pjessen> thanks, that would be great. Let's resume on the mailing list tomorrow.
 :55:27  <pjessen> A bit early still, but Merry Christmas to everone.
 :55:32  <Ada_Lovelace> I should change my password which doesn't exist.
 :56:29  <kl_eisbaer> pjessen: Merry Christmas to you :)
 :57:04  <tampakrap> pjessen: bye, merry early christmas!
 :57:38  <cboltz> so after mixing topics a bit - does someone have another status report?
 :58:34  <kl_eisbaer> heroes tickets are low as never before, yeah
 :59:40  <cboltz> I've seen that you handled quite some mirror and monitoring tickets in the last days - thanks for that!
 :00:24  <kl_eisbaer> cboltz: I hope to become two-digit until new year :-)
 :00:52  <cboltz> I'm afraid I have to disagree with "as low as never before" - we were down to ~130 in March IIRC
 :01:23  <cboltz> but I completely agree that going in the two-digit range would be very nice
 :01:29  <kl_eisbaer> now I see 129 new tickets
 :02:03  <kl_eisbaer> but anyway, I guess we just have two topics left?
 :02:04  <cboltz> strange, I see 158 open tickets
 :02:15  <cboltz> right
 :02:26  <kl_eisbaer> cboltz: I guess "open" in redmine includes "feedback" and other tickets
 :02:48  <kl_eisbaer> next meeting - Jan 2nd, or move it to Jan 9th?
 :02:58  <kl_eisbaer> I vote to the 9th
 :03:32  <tampakrap> I'm fine with both
 :03:43  <cboltz> I'm also fine with both
 :04:36  <tampakrap> so let's make it 9th since lars can't on 2
 :04:45  <kl_eisbaer> thanks!
 :04:46  <Ada_Lovelace> I'm fine with both.
 :05:10  <cboltz> ok, so the next meeting will be on Jan 9th
 :05:27  <cboltz> we already discussed the "own MX for openSUSE" topic
 :05:39  <katnip> 1400 EST ? jan 9th ?
 :05:42  <tampakrap> wait a min
 :06:00  <tampakrap> kl_eisbaer: the MX will also be relay?
 :06:14  <tampakrap> or the relay will still be the haproxy servers?
 :06:14  <kl_eisbaer> tampakrap: depends on us - or Heinlein
 :06:24  <tampakrap> ah yes I forgot heinlein
 :06:27  <kl_eisbaer> tampakrap: from my pow, it should become an own machine
 :06:55  <tampakrap> so separate pair for MX, separate pair for relay (away from haproxy)?
 :07:03  <tampakrap> (ignoring heinlein for now)
 :07:06  <kl_eisbaer> We will have some fun  how we should handle SPAM and Virus tagging, as the german law is very strict in this
 :07:21  <kl_eisbaer> tampakrap: right
 :07:31  <cboltz> oh, that's easier than you might think
 :07:48  <cboltz> completely forget about tagging, and reject the mails instead
 :07:52  <kl_eisbaer> tampakrap: maybe we should think about having mx1 in Nuremberg (or Prague?) and mx2 in Provo ?
 :08:06  <kl_eisbaer> cboltz: yes - and no
 :08:15  <cboltz> the important thing is to do the reject on the MX, so that the sending server gets a 5xx instead of a 200 to log
 :08:24  <kl_eisbaer> but I would rely on Per here, as he has the knowledge we might simple use :-)
 :08:36  <cboltz> this also means the sending server has to create the bounce mail, so we won't create backscatter
 :08:59  <kl_eisbaer> ...and of course on you, cboltz :-)
 :09:01  <tampakrap> okay
 :09:17  <kl_eisbaer> tampakrap: so we have the two admins for a possible MX already
 :09:32  <tampakrap> perfect
 :09:46  <cboltz> BTW: that's what Peer Heinlein recommends for years, and since he's an expert on mail servers _and_ studied the legal stuff, I trust him on this ;-)
 :09:58  <kl_eisbaer> I'm just not sure if we can have this (incl. the own mailboxes from Heinlein) as Christmas present for our users
 :10:33  <kl_eisbaer> for my it's just the question: who wants to drive this ?
 :11:56  <Ada_Lovelace> I told you, he is working on it...
 :12:36  <kl_eisbaer> Ada_Lovelace: who is "he" ?
 :12:43  <Ada_Lovelace> Should I forward such a email to you?
 :12:52  <Ada_Lovelace> A admin by Heinlein.
 :13:10  <kl_eisbaer> Ada_Lovelace: ah, ok. So the board already decided that Heinlein takes over our MX ?
 :13:18  <Ada_Lovelace> I received emails by mailbox.org, that my mailbox has changes.
 :13:38  <Ada_Lovelace> I should change my password for adalovelace@opensuse.org.
 :14:08  <Ada_Lovelace> Yes.
 :14:39  <kl_eisbaer> ah, ok. Because than we can forget the discussion/topic about hosting our own MX
 :14:39  <Ada_Lovelace> I don't know what Max said to Per.
 :14:58  <cboltz> the board didn't talk about the technical details (at least in the last year), but IMHO it would make sense if they do the MX
 :15:00  <kl_eisbaer> sounds like some interesting setup
 :15:51  <kl_eisbaer> Well, theoretical question, but what if people decide that their account/mail should not be given out to others ?
 :15:59  <Ada_Lovelace> That's a discussion between Max and Per.
 :16:24  <Ada_Lovelace> *at the moment*
 :16:37  <katnip> is the mail simply an alias?
 :16:38  <kl_eisbaer> wow. Congratulations, mmaher_home, for your legal exam ;-) - I hope you have enough money in your pocket ;-)
 :16:42  <mmaher_home> we asked if we can either have the forward option or creation of a mailbox inbox. still no answer
 :17:02  <Ada_Lovelace> Most email aliases are MX forwards. What will be given away?
 :17:24  <kl_eisbaer> Ada_Lovelace: user data - and might it be only the login / user names
 :18:01  <kl_eisbaer> Enough that someone at Heinlein would know that user X is an openSUSE member
 :18:16  <kl_eisbaer> even if this user never want to get a mailbox at mailbox.org
 :18:57  <kl_eisbaer> but luckily I'm not an expert in this area, so I might have a bit to much paranoia here
 :19:19  <kl_eisbaer> that's why I'm relying on board and legal decisions for such stuff
 :20:14  <kl_eisbaer> same for the DNS, btw. But luckily we don't need to think about user data here ;-)
 :21:17  <tampakrap> so moving to the last topic then?
 :21:23  <kl_eisbaer> IMHO yes.
 :21:31  <cboltz> yes
 :21:31  <Ada_Lovelace> Then you aren't allowed to host openSUSE systems at any hosting providers, because databases have user data.
 :21:59  <kl_eisbaer> I just wanted to ask if anybody would disagree if I ask legal to take over the primary NS for the opensuse.org domain
 :22:32  <tampakrap> no objections, that's one shuttle net address less :)
 :22:53  <kl_eisbaer> Ada_Lovelace: again: I'm not an expert - but I see a difference between hosting servers (and getting them maintained by openSUSE people) or directly providing data to other providers that contains user information
 :23:40  <kl_eisbaer> the only thing I have in the DNS case: we should IMHO think about running a 3rd  DNS machine somewhere else
 :23:49  <cboltz> kl_eisbaer: right, but actually we already give that data to a company - SUSE ;-)
 :24:07  <kl_eisbaer> cboltz: yes, and everybody agreed during account creation that this is ok
 :24:21  <kl_eisbaer> ...and everybody need to agree again if you give the data to another company
 :24:54  <kl_eisbaer> but again: that's just my personal understanding and I'm no lawyer
 :25:04  <kl_eisbaer> so I'm fine with whatever the board decides
 :25:28  <cboltz> feel free to talk to legal and/or to Richard about this ;-)
 :25:35  <Ada_Lovelace> Per is lawyer, too.
 :25:36  <kl_eisbaer> The good thing, if we run our own DNS would be: we could provide Geo-based DNS :-))
 :26:20  <kl_eisbaer> again: I don't want to object here - I was just surprised that this topic is already in that state where Heinlein checks the setup
 :26:55  <kl_eisbaer> I'm happy that something goes forward here - so go ahead and drive it :-)
 :26:55  <tampakrap> regarding the third dns server, that's a big blocker
 :27:31  <kl_eisbaer> yes / no - it depends. We have of course enough free IPs to host it either in Nuermberg or Provo
 :27:49  <kl_eisbaer> and I'm sure we will find enough "DNS masters" who will include our domain as slave
 :28:03  <tampakrap> true
 :28:17  <kl_eisbaer> but if we want to go further (the Geo-Based thing), we should not take this too easy
 :28:45  <kl_eisbaer> IMHO starting with a basis setup with 2 servers in NUE and 2 in Provo should be totally ok
 :29:28  <kl_eisbaer> once we are able to run a "useful" amount of services also in Provo (like wiki, download.o.o. or such), we should think about the geo-base DNS again and enroll that
 :29:50  <kl_eisbaer> as it would improve the availability and usability for our users, especially in the US
 :29:53  <tampakrap> we need to make the vpn in provo first and figure out how to bridge the vpn servers
 :29:59  <kl_eisbaer> ...and maybe even Asia
 :30:01  <tampakrap> right?
 :30:22  <kl_eisbaer> tampakrap: yes, this is meanwhile one of the hottest topics for me after the download.o.o switch
 :30:58  <kl_eisbaer> tampakrap: if scar would open a VPN tunnel to the machines in Provo, it could route the "default gateway" traffic already
 :31:17  <kl_eisbaer> tampakrap: only the machines with more than one interface need to get an additional routing entry for the Provo network range
 :31:36  <tampakrap> okay
 :32:50  <Ada_Lovelace> DNS servers can be everywhere on the world and can communicat together the.
 :32:59  <kl_eisbaer> ok - any other questions?
 :33:07  <Ada_Lovelace> *communicate together then*
 :33:19  <kl_eisbaer> Ada_Lovelace: I would even say: they *should* be everywhere in the world :-)
 :33:50  <kl_eisbaer> ...and establishing DNSSec is also a good idea, IMHO ;_)
 :34:04  <Ada_Lovelace> I know colleages who wanted to have their own DNS servers so for HA.
 :38:30  <kl_eisbaer> any other points ?
 :38:34  <cboltz> kl_eisbaer: willl you setup all those DNS servers manually, or will you use salt from the beginning this time? ;-)
 :38:57  <kl_eisbaer> cboltz: I doubt I will have enough time for this
 :39:03  <cboltz> hint: seeing that you want at least 4 servers, salt is probably faster
 :39:47  <kl_eisbaer> cboltz: on the other way: DNS is really easy
 :40:15  <kl_eisbaer> as freeipa will stay as master, all the other will get their information from freeipa anyway
 :40:52  <kl_eisbaer> cboltz: ...and in this case Salt can't be faster than a "dd" of a golden image ;-p
 :41:18  <tampakrap> the dns config of chip is already in salt btw
 :41:28  <cboltz> if you have to copy that image to Provo, things might be different ;-)
 :41:29  <kl_eisbaer> cboltz: ...and you might think about other admins who will just include the opensuse.org domain as additional slave zone
 :42:08  <kl_eisbaer> I'm not sure if they would give your salt master access to their servers if you are speaking about two zones only
 :42:16  <kl_eisbaer> ah, autsch!
 :42:28  <kl_eisbaer> found an important bug in my thinking :-/
 :42:45  <cboltz> ;-)
 :42:47  <kl_eisbaer> at least in Nuremberg, the reverse zone is a problem
 :43:06  <tampakrap> ah yes
 :43:08  <kl_eisbaer> something I need to clarify with MF-IT
 :43:51  <cboltz> so in worst case reverse lookups would still need the MF DNS servers?
 :44:05  <kl_eisbaer> cboltz: ...and that is something that will not work
 :44:28  <kl_eisbaer> ...but I'm sure we will find a solution
 :44:29  <cboltz> silly question - why not?
 :45:04  <cboltz> my understanding is that having separate DNS servers for the reverse lookup shouldn't be a problem
 :45:28  <kl_eisbaer> yeah - but it does not help with the separation
 :45:52  <kl_eisbaer> if I still need to open ticket after ticket for every single IP, that simply just boring
 :46:07  <Ada_Lovelace> I have to leave you. Tomorrow is university.
 :46:26  <kl_eisbaer> Ada_Lovelace: have a lot of fun there :-)
 :46:28  <tampakrap> Ada_Lovelace: have a nice evening
 :46:29  <Ada_Lovelace> Bye. Until next meeting.
 :46:34  <cboltz> bye
 :46:53  <kl_eisbaer> let me start the discussion with MF-IT how we can solve that finally.
 :47:21  <cboltz> I get your point and agree that the perfect solution would be to also handle the reverse lookups ourself
 :48:04  <cboltz> but even if this doesn't work, handling the forward DNS (aka what people use 99,9% of the time) would be a big step forward
 :48:32  <kl_eisbaer> cboltz: depends - I'm not sure if this will work for geo-based DNS
 :48:44  <kl_eisbaer> ...and for implementing DNSSec this does also not really help
 :49:22  <kl_eisbaer> but anyway - I'll take that with me for now
 :49:36  <kl_eisbaer> at least it's good to hear that nobody has objections against it
 :50:50  <cboltz> I'm quite sure nobody wants to depend on MF-IT if it can be avoided ;-)
 :51:03  <kl_eisbaer> ok - anything left to discuss ?
 :51:52  <cboltz> nothing from me
 :52:02  <tampakrap> nope, we can discuss further with merge requests https://gitlab.infra.opensuse.org/infra/salt/merge_requests :)
 :52:23  <kl_eisbaer> ok - in this ^^ case it's time for me to leave  ;-)
 :52:40  <kl_eisbaer> have a good night!
 :52:45  <tampakrap> good night!
 :52:48  <katnip> see ya
 :52:49  <cboltz> good night!
 :53:00  <cboltz> and thanks everybody for staying so long!
 :53:10  <katnip> np

(1-1/1)

Project

General

Profile

openSUSE admin

communication #27500 » 2017-12-05-heroes-meeting.txt