openSUSE admin

2017-11-07 heroes meeting

[20:05:04] <cboltz> so - welcome everybody to the monthly Heroes meeting!

[20:05:19] <cboltz> does someone from the community have questions?

[20:05:33] <tampakrap> I'm moving the topics from last meeting

[20:06:29] <tampakrap> added

[20:07:02] <cboltz> looks like nobody has questions, so let's move to the status reports

[20:07:38] <kl_eisbaer> who should start?

[20:08:11] <cboltz> what about you? ;-)

[20:08:17] <kl_eisbaer> :D

[20:08:40] <kl_eisbaer> Monitoring: we are now at 40 VMs and >900 monitored services

[20:08:41] <Ada_Lovelace> You can tell us a lot. :)

[20:08:59] <tampakrap> wow

[20:09:20] <kl_eisbaer> as I wrote in the news, not only the checks are interesting, also the information provided via the graphs are helpful

[20:09:39] <kl_eisbaer> ...so - as example - to see how many people are logged in via VPN all the time ;-)

[20:10:07] <kl_eisbaer> I plan to get more information also from our nginx and lighttpd instances, but this seems to be a bit tricky

[20:10:49] <kl_eisbaer> JFYI: I disabled notifications for the Updates check, as this turned out to be the service with most notifications ;-)

[20:11:17] <kl_eisbaer> ..and as we update on Thursday anyway, I think it is ok to disable notifications for this check

[20:12:03] <kl_eisbaer> I also wrote a short "run_zypper_up" script, which is in /root/bin/ on monitor.infra.opensuse.org

[20:12:14] <pjessen> I'm back

[20:12:17] <cboltz> would it be possible to run the update check _after_ the cronjob that auto-installs most updates?

[20:12:40] <kl_eisbaer> if you log in to this machine via SSH and have agent forwarding enabled, running this script helps to automate the maintenance on thursday

[20:12:55] <kl_eisbaer> cboltz: the update check *is* running after that, yes

[20:13:40] <cboltz> :-)

[20:13:47] <kl_eisbaer> Other topic:

[20:13:57] <kl_eisbaer> The Provo mirror supports now http2 protocol

[20:14:18] <kl_eisbaer> ...but we need to investigate if libcurl uses that protocol automatically or not

[20:15:15] <kl_eisbaer> I talked with the zypper maintainer, and he told me that it might be possible to add a configurable feature, it needed. So zypper could make use of the http2 protocol, too

[20:15:30] <kl_eisbaer> tcp fast open is enabled anyway already

[20:16:07] <kl_eisbaer> The Nuremberg mirror (aka download.opensuse.org) is getting closer to a state where I can open it up to the heroes:

[20:16:23] <kl_eisbaer> SUSE specific stuff has been migrated to another machine already

[20:16:47] <kl_eisbaer> so what's left is to have the machine re-installed with openSUSE and place it in the heroes network

[20:17:05] <pjessen> sounds good

[20:17:14] <kl_eisbaer> we might need a special network interface for the push from the OBS, but that should not be a big problem

[20:17:24] <cboltz> will that re-install happen with salt? ;-)

[20:17:34] <kl_eisbaer> I hope to get this done next week

[20:17:49] <kl_eisbaer> cboltz: problably not - as long as I do it

[20:18:12] <kl_eisbaer> Next mirror under our control: widehat, aka rsync.opensuse.org

[20:18:24] <kl_eisbaer> this host as currently a major problem: disk full

[20:18:51] <kl_eisbaer> as we can not extend the storage of this machine, I'm currently thinking to declare our mirror in Provo as new rsync.opensuse.org

[20:19:19] <kl_eisbaer> as this mirror in Provo is currently the only one who has everything that is also available on download.opensuse.org

[20:19:36] <kl_eisbaer> => in the end it's just about the DNS name

[20:19:40] <Ada_Lovelace> Is this mirror working correctly now?

[20:20:00] <kl_eisbaer> so far no problem reports so far (other than one bug report about the used style sheet)

[20:20:06] <Ada_Lovelace> I know about problems with the provo mirror in the past...

[20:20:11] <tampakrap> it doesn't have ipv6 though, would that be a problem?

[20:20:17] <kl_eisbaer> Ada_Lovelace: I guess the problem in the past was just a missing redirector setup

[20:20:41] <kl_eisbaer> tampakrap: right, but latest rumor has it that MF-IT is able now to assign IPv6 addresses in Provo, too

[20:20:50] <tampakrap> ah nice

[20:20:53] <Ada_Lovelace> nice

[20:21:15] <kl_eisbaer> tampakrap: so let's wait for my request for IPv6 addresses, check of the rsync modules (and the "knapsack" stuff) to work

[20:21:24] <kl_eisbaer> and then switch over

[20:22:05] <kl_eisbaer> I still need to get some approval because of the additional bandwidth used, but I see currently not really any other chance to provide a fully equipped rsync.opensuse.org

[20:22:41] <kl_eisbaer> ...or we find a mirror as "sponsor" that hosts all the stuff, including the knapsack module, for us

[20:23:16] <kl_eisbaer> JFYI: the python-knapsack scripts fill up the 80g, 160g, 320g, ... modules with content based on the access logs

[20:23:43] <cboltz> what exactly does " we can not extend the storage" mean? All disks in NBG full, or just a restriction on this VM?

[20:23:59] <kl_eisbaer> The widehat machine is not running inside the Nuremberg office

[20:24:30] <kl_eisbaer> cboltz: widehats place is sponsored by QSC - and is an old machine with "just 7TB" capacity

[20:24:59] <kl_eisbaer> cboltz: does that answer your question?

[20:25:02] <Ada_Lovelace> I know the new hosting boss at QSC. He changed from 1&1 to them after my election.

[20:25:05] <cboltz> yes, thanks

[20:25:08] <Ada_Lovelace> Should I write him?

[20:25:25] <kl_eisbaer> Ada_Lovelace: in the past, our problem was that they want to end the sponsoring ...

[20:25:28] <Ada_Lovelace> That's the ex 1&1 hosting boss.

[20:25:35] <Ada_Lovelace> Oh...

[20:25:45] <kl_eisbaer> ...so we tried ot avoid any requests ...

[20:26:00] <pjessen> I guess it's more about bandwidth than the actual disk space?

[20:26:02] <kl_eisbaer> as long as the machine is up and running, everything is ok for us

[20:26:35] <kl_eisbaer> pjessen: in the past, widehat was indeed a bandwidth saver for the NUE office

[20:26:48] <kl_eisbaer> especially as we had a dark fiber to their data center

[20:27:09] <kl_eisbaer> so we could push stuff from download.o.o to widehat via that dark fiber

[20:27:31] <kl_eisbaer> ...and everybody else was downloading from widehat using the bandwidth from QSC

[20:27:39] <pjessen> yeah, I get the picture.

[20:27:56] <kl_eisbaer> the machine is an old system with a local RAID controller

[20:28:11] <kl_eisbaer> the maximum amount of disks is installed

[20:28:20] <kl_eisbaer> ...and it sadly supports only 1TB disks

[20:28:28] <kl_eisbaer> so extending it is not possible

[20:28:54] <kl_eisbaer> we might be able to replace the machine silently with a new one with bigger disks, but so far I don't see that happen

[20:29:45] <pjessen> how about a bigger raid controller? I have some spare that will take 2Tb disks.

[20:30:04] <kl_eisbaer> might be possible, but I need to check the hardware first

[20:30:23] <Ada_Lovelace> You don't have access to the data center?

[20:30:35] <kl_eisbaer> Ada_Lovelace: I have access, yes.

[20:30:53] <kl_eisbaer> But I need to fire up the next VPN to get access to the machine for checking dmidecode ;-)

[20:31:40] <kl_eisbaer> => what about moving the discussion about widehat to "later", resp. mailing list, once I have the hardware details ?

[20:32:07] <cboltz> sounds like a good idea

[20:32:09] <pjessen> yep.

[20:32:16] <kl_eisbaer> Next topic: logs from #opensuse-admin

[20:32:36] <kl_eisbaer> It's true, I'm lazy ;-)

[20:32:56] <kl_eisbaer> ...and I'm not running my IRC client all the day, but our bot is there...

[20:33:23] <kl_eisbaer> ...so I - in my glory ;-) decided to let it log for me, but I guess this might be useful for others, too?

[20:33:31] <kl_eisbaer> => https://monitor.opensuse.org/heroes/

[20:33:55] <kl_eisbaer> I could put the URL behind LDAP auth, but at the moment, I think it might be useful also for others.

[20:34:01] <kl_eisbaer> => what do you think?

[20:34:11] <tampakrap> leave it public, it's a public channel after all

[20:34:18] <Ada_Lovelace> It should be public...

[20:34:26] <cboltz> yes, keep it public

[20:34:41] <kl_eisbaer> fine with me ;-)

[20:34:47] <cboltz> but maybe add a note to /topic to make people aware that the channel is logged

[20:34:58] <kl_eisbaer> cboltz: up to you ;-)

[20:35:15] <kl_eisbaer> we might also put a link to the log in our wiki page ?

[20:35:24] <Ada_Lovelace> But we should speak about the time how long it should be saved or when cleaned.

[20:35:33] <cboltz> I doubt I have permissions to update /topic

[20:35:55] <kl_eisbaer> cboltz: IMHO /op should work in this channel for everyone

[20:36:00] <tampakrap> +1 for the link to the wiki page

[20:36:13] <kl_eisbaer> Ada_Lovelace: any suggestions for the time frame?

[20:36:13] [Fehler] Sie müssen in #opensuse-admin Operator sein, um das tun zu können.

[20:36:52] <cboltz> kl_eisbaer: no, /op gives me "you have to be operator in #opensuse-admin to do that"

[20:36:53] <Ada_Lovelace> 1 - 2 months (depends on the size of the disk)

[20:37:19] <kl_eisbaer> cboltz: sorry, I mean you should ask chanserv to become op

[20:37:41] <kl_eisbaer> cboltz: if you have a suggestion for the topic header, I can do that later for you ;-)

[20:37:53] <kl_eisbaer> Ada_Lovelace: the size does not really matter

[20:38:35] <kl_eisbaer> Ada_Lovelace: we are speaking about 472k at the moment

[20:38:45] <cboltz> I'd vote to keep the logs forever like we do with ML archives

[20:38:46] <Ada_Lovelace> Then 3 months should be more than enough... It can be that anybody wants to refer to any task in the chat.

[20:39:09] <cboltz> deleting something from the internet doesn't work anyway ;-)

[20:39:26] <kl_eisbaer> cboltz: +1 from my side

[20:39:32] <Ada_Lovelace> Ok

[20:39:41] <tampakrap> restrict the search engines maybe?

[20:39:41] <kl_eisbaer> if we need to censor something, we can do it on the monitoring host anyway

[20:39:51] <kl_eisbaer> tampakrap: fine with me

[20:40:21] <tampakrap> okay

[20:40:33] <tampakrap> the bot is amazing, thanks a lot for it!

[20:40:54] <kl_eisbaer> tampakrap: you can even enhance it to a "meet bot" ;-)

[20:41:14] <tampakrap> where is it running btw? on scar?

[20:41:24] <kl_eisbaer> tampakrap: ...or connect it to the rabbitmq queue and get informations pushed here once a package in openSUSE:infrastructure is built

[20:41:34] <kl_eisbaer> tampakrap: it's running on the monitor machine ;-)

[20:41:35] <tampakrap> yeah that would be amazing

[20:41:55] <tampakrap> or to send new merge requests and commits in the salt repository

[20:42:22] <kl_eisbaer> tampakrap: ...and all you need for this is netcat ;-)

[20:42:57] <kl_eisbaer> I've placed an example script in /root/bin/send_irc_message on the monitor machine

[20:42:57] <kl_eisbaer>

[20:43:09] <kl_eisbaer> ...if you want to try it out.

[20:43:33] <kl_eisbaer> At the moment, the port is bond to localhost, but I can open it up to the whole infra.opensuse.org network, if this is needed

[20:44:27] <kl_eisbaer> Other topic: galera cluster ?

[20:44:28] <heroes-bot> PROBLEM: MySQL WSREP recv on galera2.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 1.184783 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera2.infra.opensuse.org&service=MySQL%20WSREP%20recv

[20:44:44] <kl_eisbaer> ...no, that ^^ was not planned ;-)

[20:44:54] <cboltz> I already wondered ;-)

[20:44:56] <tampakrap> hahahaha

[20:45:06] <kl_eisbaer> but as you can see, the cluster still needs some fine tuning

[20:45:30] <kl_eisbaer> I just took some default values and combined them with some "good practices" from our internal cluster at SUSE

[20:45:56] <kl_eisbaer> At the moment, I have 3 interesting things:

[20:46:36] <kl_eisbaer> * mysql-tuning-scripts is updated and contains now the latest mysqltuner.pl script, which is analizing a running mysql instance and (new) also gives some hints for galera cluster

[20:47:30] <kl_eisbaer> * I will update/change the VM definition for the machines, to allow more features from the hypervisor CPUs - sadly this requires a complete power cycle of the VM

[20:47:46] <kl_eisbaer> ^^^ that should give hopefully a bit more performance

[20:48:24] <kl_eisbaer> * I think we don't need to migrate so many databases as I initially thought: most of the DBs hosted at the old cluster are meanwhile obsolete

[20:48:51] <kl_eisbaer> The biggest DB will be the one from beans.o.o aka piwik

[20:49:32] <kl_eisbaer> I'm currently thinking to add the new cluster as slave for this DB to the old cluster, to avoid a big downtime if we migrate piwik

[20:49:53] <kl_eisbaer> but I haven't done that, so I need to do some more testing

[20:50:24] <kl_eisbaer> the other databases (incl. the wiki ones) should be "ready to migrate" in one or two weeks, IMHO

[20:51:02] <kl_eisbaer> if someone is interested and wants to become a "DB-Admin", I'm happy to hand over ;-)

[20:51:15] <cboltz> just tell me when you want to migrate the wiki DBs, so that we can make the wikis read-only during that time

[20:51:26] <kl_eisbaer> cboltz: will do, of course

[20:51:50] <cboltz> well, ideally I'd like to hand over creating database users and databases to salt ;-)

[20:51:59] <kl_eisbaer> I hope to have just minimal impact for all DBs other than the piwik one

[20:52:21] <kl_eisbaer> cboltz: but even than, you definitively want to have some DB-Admin who cares about your profile ;-)

[20:52:45] <kl_eisbaer> otherwise, your wikis would suddenly become very, very slow ....

[20:53:17] <kl_eisbaer> ...another topic: svn/kernel.opensuse.org

[20:53:21] <cboltz> I know salt can't do automated performance monitoring and tuning ;-)

[20:53:44] <kl_eisbaer> the current plan is to migrate the machine hosting the two services on Thursday to the new network

[20:54:26] <kl_eisbaer> ...that's all I have as "status report" so far

[20:55:23] <tampakrap> one quick question

[20:55:33] <tampakrap> the mysql servers are using the official opensuse packages?

[20:55:42] <kl_eisbaer> no

[20:55:55] <tampakrap> okay

[20:56:02] <kl_eisbaer> they are using the ones from server:database, as the latest galera features are very fresh

[20:56:22] <kl_eisbaer> the ones for 42.3 don't support galera so good, (yet)

[20:56:53] <kl_eisbaer> with the latest packages from server:database, it's more or less just adding the galera packages and a configuration snipplet

[20:56:59] <tampakrap> got it

[20:57:18] <kl_eisbaer> any other status reports ?

[20:57:36] <pjessen> no much from me, havent had time since the summer holidays

[20:57:41] <tampakrap> about salt

[20:57:51] <tampakrap> we support now gpg encrypted pillars

[20:57:55] <kl_eisbaer> pjessen: is the "sience" list online already ?

[20:58:05] <pjessen> it should be, yes.

[20:58:12] <pjessen> opensuse-science@o.o

[20:58:21] <kl_eisbaer> ok, thanks

[20:58:45] <kl_eisbaer> tampakrap: what does this mean ?

[20:59:10] <tampakrap> this means that we can put passwords in pillars

[20:59:30] <kl_eisbaer> nice :-)

[20:59:42] <tampakrap> there is a MR open that has the passwords for the keepalived config on anna/elsa encrypted

[21:00:03] <tampakrap> it needs documentation though on how to use it and about the structure

[21:00:09] <tampakrap> and then I'll proceed with that MR

[21:00:43] <kl_eisbaer> :D

[21:01:13] <kl_eisbaer> BTW: daffy (aka login2) is prepared for the 2nd daffy already - I setup the keepalived there already

[21:02:03] <tampakrap> cool

[21:02:08] <kl_eisbaer> BTW2: something for cboltz (-: https://monitor.opensuse.org/pnp4nagios/index.php/graph?host=redmine.infra.opensuse.org&srv=Heroes+tickets&view=2

[21:02:46] <cboltz> yes, I already noticed that :-)

[21:02:52] <kl_eisbaer> ^^ should give you a quick overview about the current tickets on progress.opensuse.org

[21:03:24] <kl_eisbaer> I also defined some warning/critical levels - just to create some fun here in the channel ;-)

[21:03:30] <pjessen> hehe

[21:04:46] <kl_eisbaer> but while I was creating that check, I was wondering if we shouldn't define a maximum lifetime for a ticket

[21:05:25] <kl_eisbaer> I was just wondering if tickets that are older than a year are really interesting anyone any longer?

[21:05:26] <Ada_Lovelace> We had the same in Bugzilla...

[21:06:09] <kl_eisbaer> Ada_Lovelace: ...and was there a solution ?

[21:06:11] <Ada_Lovelace> Christian and I found some tickets from the past which were interesting for us.

[21:06:17] <pjessen> in bugzilla, things take time.

[21:06:29] <Ada_Lovelace> Reviewing and pinging was the solution.

[21:07:24] <kl_eisbaer> maybe we should schedule a "progress" cleanup event ?

[21:07:28] <cboltz> right, let's handle tickets quickly instead of inventing an auto-close (which is more or less a motivation to be lazy IMHO)

[21:07:47] <pjessen> agree.

[21:07:49] <cboltz> yes, such a cleanup would make sense

[21:08:20] <kl_eisbaer> maybe a good transfer to the next topic: " offsite meeting? " :-)

[21:08:28] <Ada_Lovelace> I would be available.

[21:09:05] <cboltz> kl_eisbaer: indeed, pinging people in the same room is much easier ;-)

[21:09:24] <kl_eisbaer> any suggestions ?

[21:09:41] <kl_eisbaer> ...I hope everyone knows that SUSE will have the yearly hackweek starting on Friday ?

[21:09:41] <pjessen> somewhere in Zurich?

[21:09:47] <pjessen> :-)

[21:09:55] <kl_eisbaer> pjessen: fine with me :-)

[21:09:58] <Ada_Lovelace> I saw it today.

[21:10:14] <kl_eisbaer> pjessen: but I guess you need to organize the "where" in Zurich for us :-)

[21:10:29] <tampakrap> zurich is fine for me, my brother lives there

[21:10:50] <kl_eisbaer> heya: so we have a Party location already :-)

[21:12:20] <Ada_Lovelace> But wait... Are trains from Germany to Zurich going at the moment? There is a lot damageg, if I think back.

[21:12:37] <Ada_Lovelace> *damaged*

[21:12:39] <kl_eisbaer> pjessen: can you try to organize something ?

[21:12:43] <pjessen> I think that tunnel was fixed already.

[21:12:54] <Ada_Lovelace> Really? Then all is ok.

[21:13:14] <pjessen> I'll have to say no - at least not on this side of Christmas

[21:13:17] <kl_eisbaer> pjessen: I guess the main need is a reliable internet connection

[21:13:41] <kl_eisbaer> pjessen: next year should be not a big problem ;-)

[21:14:22] <pjessen> What time frame do we have in mind? Roughly.

[21:14:23] <Ada_Lovelace> February should be the best for me because of semester holidays. ;-)

[21:14:43] <pjessen> Yeah, we have Sportferien in Feb too.

[21:15:04] <tampakrap> fine by me

[21:15:09] <cboltz> as long as you avoid the carnival and FOSDEM weekends, February sounds good

[21:15:20] <kl_eisbaer> I guess 2-3 days (a weekend?) should really be enough

[21:15:44] <Ada_Lovelace> We weekend after FOSDEM is good.

[21:16:01] <pjessen> Is this something we should continue on the mailing list later?

[21:16:18] <kl_eisbaer> pjessen: jip, good idea

[21:16:24] <cboltz> no, the weekend after FOSDEM is carnival

[21:16:27] <tampakrap> perfect

[21:16:40] <Ada_Lovelace> Then we can have a party. ;-)

[21:16:42] <cboltz> and I know ~40 people who would hate me if I refuse to drive the carnival float ;-)

[21:17:19] <Ada_Lovelace> Let's speak about it on the mailinglist.

[21:17:21] <kl_eisbaer> So it looks like everybody is looking forward to a meeting in Zurich :-)

[21:17:34] <pjessen> Wow, I should have kept my mouht shut ....

[21:17:51] <kl_eisbaer> otherwise we could always schedule a meeting here in Nuremberg again. That should not really be a problem.

[21:18:04] * kl_eisbaer now waits for Theo to invite everyone to Prague ;-)

[21:18:11] <tampakrap> prague yey!

[21:18:19] <tampakrap> we have the conference as well here

[21:18:31] <Ada_Lovelace> Yes. The oSC

[21:18:37] <tampakrap> or actually, let's go to greece!

[21:18:38] <pjessen> I don't mind setting something up, but Nuernberg is within range for me too.

[21:18:53] <pjessen> Too cold there in Feb :-)

[21:19:18] <kl_eisbaer> pjessen: I just don't wanted to put pressure on you, that's why I offered NUE again ;-)

[21:19:56] <kl_eisbaer> pjessen: but we can of course include a survival training in the offsite meeting

[21:20:16] <kl_eisbaer> ^ => let's move the discussion to the mailing list ;-)

[21:20:21] <pjessen> thanks - I have a lot on my plate, to be honest.

[21:20:41] <pjessen> ok

[21:20:53] <kl_eisbaer> Next topic: enhance infra.opensuse.org domain ?

[21:21:05] <kl_eisbaer> that one's from me

[21:21:18] <pjessen> "enhance" ?

[21:21:38] <kl_eisbaer> in short: are there any objections, if I add the "service" names as aliases to our hosts with their special names?

[21:21:58] <kl_eisbaer> boosters.infra.opensuse.org would have an additional alias connect.infra.opensuse.org as example

[21:22:14] <Ada_Lovelace> infra.opensuse.org isn't the right thing for real domainnames.

[21:22:24] <Ada_Lovelace> That's something for hostnames.

[21:22:26] <kl_eisbaer> that would make it a bit easier - at least for me - to get to the "right" machine directly

[21:22:37] <tampakrap> I would like to have them, and I created some as well already

[21:22:38] <pjessen> Sounds good, that's what we do with all services locally. A service can always move.,

[21:22:40] <kl_eisbaer> Ada_Lovelace: ?

[21:23:12] <Ada_Lovelace> You want to offer easy domainnames for users without infra in it.

[21:23:26] <kl_eisbaer> Ada_Lovelace: no, sorry

[21:23:28] <Ada_Lovelace> I know such names only as hostnames.

[21:23:39] <kl_eisbaer> Ada_Lovelace: I just want to make my live as admin easier

[21:23:41] <pjessen> it's just a dns cname

[21:23:56] <tampakrap> no, the point is to be able to do `ssh connect` instead of going to the machine list to find out where you need to ssh to check what is broken on connect

[21:24:08] <Ada_Lovelace> If we have the cname additionally to this name, then ok.

[21:24:09] <tampakrap> CNAMEs are cheap

[21:24:10] <kl_eisbaer> tampakrap: exactly

[21:24:15] <tampakrap> are for free actually

[21:24:22] <Ada_Lovelace> I don't like the situation with gitlab.

[21:24:27] <tampakrap> totally desired imho

[21:24:48] <tampakrap> gitlab moved from gitlab.o.o to gitlab.infra.o.o

[21:24:51] <kl_eisbaer> Ada_Lovelace: that's not related to my request, sorry

[21:25:00] <Ada_Lovelace> ok

[21:25:05] <kl_eisbaer> I'm speaking about all the machines like scar.infra.opensuse.org

[21:25:24] <kl_eisbaer> not everyone knows what scar.infra.opensuse.org or mickey.infra.opensuse.org is doing

[21:25:58] <pjessen> adding service names is a good thing.

[21:26:02] <kl_eisbaer> but if they become aliases like "vpn.infra.opensuse.org" or "salt.infra.opensuse.org", most people might instantly know which services they can find on those machines

[21:26:40] <kl_eisbaer> Ada_Lovelace: does this make it a bit clearer to you?

[21:27:26] <Ada_Lovelace> Yes

[21:27:33] <kl_eisbaer> still objections ?

[21:28:25] <Ada_Lovelace> I like service names as additional names.

[21:28:47] <kl_eisbaer> ok - I take this as a "yes" from everyone ;-)

[21:28:52] <kl_eisbaer> thanks!

[21:29:10] <kl_eisbaer> Next topic: sponsoring offer from cPanal (see Doug's mail on the ML)

[21:29:24] <tampakrap> typo, I'll fix it

[21:30:03] <kl_eisbaer> I've heard no news about this topic - anyone else?

[21:30:26] <pjessen> nothing

[21:30:45] <kl_eisbaer> anyone who wants to drive this ?

[21:31:07] <tampakrap> I will ask Max tomorrow, I *think* he communicated something about htis already

[21:31:10] <tampakrap> so AI for me

[21:31:21] <kl_eisbaer> thanks

[21:31:50] <kl_eisbaer> Next topic: transfer opensuse.cz domain (another mail from Doug)

[21:32:01] <tampakrap> I object on this

[21:32:11] <kl_eisbaer> tampakrap: ok

[21:32:42] <tampakrap> I told Petr (the original requestor) already that I don't like the idea that the official opensuse DNS will handle domains from other community teams

[21:32:54] <tampakrap> their request is simple though, they want a redirect to the wiki

[21:33:11] <tampakrap> but we will have to accept other domains in the future as well, and maintain them

[21:33:16] <tampakrap> do we want to do that?

[21:33:39] <Ada_Lovelace> Why not?

[21:33:45] <pjessen> Who is behind opensuse.cz ?

[21:33:48] <cboltz> if it's just a redirect, it's easy to answer 'yes'

[21:34:17] <cboltz> besides that, having control over opensuse.* domains can't hurt

[21:34:17] <kl_eisbaer> tampakrap: just to understand you right: Petr offered us to get the owner of the opensuse.cz domain?

[21:35:02] <pjessen> can "we" even act as the owner?

[21:35:10] <tampakrap> correct, I said that I will take the topic to our meeting, but he went as well to doug, who sent the mail first to our ml

[21:35:47] <kl_eisbaer> from a technical standpoint, I see no issues - but the legal point needs to be clarified by the board IMHO

[21:36:08] <cboltz> actually it's already on the board's radar

[21:36:35] <cboltz> last I heard is that Richard waits for response from Ciaran who should be in the whois data

[21:37:04] <tampakrap> okay mind responding to Petr and Doug then?

[21:37:11] <pjessen> the whois data will accept anything.

[21:37:28] <kl_eisbaer> pjessen: :-)

[21:37:47] <cboltz> pjessen: you are technically right, but legally Ciaran might have a different opinion ;-)

[21:37:47] <kl_eisbaer> tampakrap: I would say: go ahead

[21:38:04] <kl_eisbaer> ...as we are just the technical part of the story

[21:38:15] <kl_eisbaer> tampakrap: but what is with their webside?

[21:38:43] <tampakrap> last thing they told me, they want to get rid of it

[21:38:47] <kl_eisbaer> I guess they want to leave that stuff as it is and just want to get the opensuse.cz domain (DNS) under openSUSE control?

[21:39:25] <pjessen> there is presuambly also an issue of cost?

[21:39:30] <kl_eisbaer> tampakrap: maybe put this (together with the Email question) in your answer email?

[21:39:39] <tampakrap> ack

[21:40:22] <kl_eisbaer> means: is it ok for them that openSUSE takes over the domain, redirects anything opensuse.cz related to the CZ wiki and skip the email part?

[21:40:31] <kl_eisbaer> ...something like that

[21:41:08] <kl_eisbaer> ok for everyone ?

[21:41:26] <cboltz> yes

[21:41:29] <pjessen> So we would be just bne the DNS admns?

[21:42:00] <Ada_Lovelace> yes

[21:42:20] <kl_eisbaer> pjessen: IMHO yes. That would be my understanding

[21:42:37] <pjessen> I have no issue with that. Especially as I don't do any DNS admin ...

[21:42:53] <kl_eisbaer> pjessen: not yet... ;-)

[21:43:16] <kl_eisbaer> I guess the next 2 topics were handled already: monitoring/mirror status ?

[21:43:33] <Ada_Lovelace> yes. You told all. ;)

[21:43:39] <tampakrap> and the salt status

[21:44:01] <kl_eisbaer> tampakrap: I'm happy to hear more ;-)

[21:45:16] * cboltz wonders how the 40 VMs match the 28 pillar/id/* files

[21:45:40] <kl_eisbaer> cboltz: now you know how many machines are administrated by me ;-)

[21:46:00] <cboltz> lol

[21:46:15] <tampakrap> yep let's fix that

[21:46:18] <kl_eisbaer> at least the galera machines are currently completely unmanaged

[21:46:18] <cboltz> but seriously - why don't you use salt?

[21:46:43] <kl_eisbaer> cboltz: fear? not enough knowledge? using ansible ?

[21:46:51] <kl_eisbaer> cboltz: choose one and you might be right

[21:47:51] <cboltz> I can help you to fix #2, which might then also fix #1 ;-)

[21:48:08] <kl_eisbaer> cboltz: thanks - I will definitively come back to that :-)

[21:48:23] <kl_eisbaer> tampakrap: btw, I've one question

[21:48:33] <tampakrap> shoot

[21:48:46] <kl_eisbaer> tampakrap: is the salt master in the heroes network only serving for the heroes machines ?

[21:48:59] <tampakrap> yes

[21:49:06] <kl_eisbaer> I was just wondering as I was on the svn machine in the other network

[21:49:26] <kl_eisbaer> but that might be just a "flash back", as I saw there some repos pointing to "nowhere"

[21:49:39] <tampakrap> we used to have a separate master for the suse-dmz but it is currently broken since the network split

[21:49:49] <kl_eisbaer> ok

[21:49:57] <tampakrap> hopefully will be back during hackweek

[21:50:13] <kl_eisbaer> about the ntp stuff - I've two notes from what I saw so far...

[21:50:22] <Ada_Lovelace> That's your hackweek project? :D

[21:50:26] <kl_eisbaer> 1) any objections using chrony instead of ntpd ?

[21:50:46] <pjessen> Any reason why?

[21:50:58] <kl_eisbaer> 2) any objections to use just "ntp1", "ntp2" instead of the full DNS name?

[21:51:05] <tampakrap> no objections, I saw the configs are quite similar, but I'll need to disable it in salt as well, as ntp is currently managed in salt

[21:51:24] <kl_eisbaer> chrony is a bit more secure, as some tests from secint showed

[21:51:44] <tampakrap> for 2 it will work if the machines have properly set up the searchlist right?

[21:51:46] <kl_eisbaer> the chrony maintainers did not implement all features, so they are not fully RFC compatible

[21:51:52] <cboltz> tampakrap: sounds like kl_eisbaer found a nice task to practise salt ;-)

[21:52:05] <kl_eisbaer> but they implemented enough for having everything that a client needs

[21:52:27] <pjessen> I'm a long-time fan of ntp, but that's personal.

[21:52:59] <kl_eisbaer> one example: chrony only binds to localhost and no other interface per default

[21:53:15] <kl_eisbaer> pjessen: I was, too, but times are changing ;-)

[21:53:56] <kl_eisbaer> tampakrap: yes, with just the hostname (ntp1), they will always just try their local domain

[21:54:37] <kl_eisbaer> pjessen: I just found one more or less important feature that chrony handles different than ntpd

[21:55:20] <kl_eisbaer> the "tinker panic 0" (which is btw. missing in the salt profile), is translated there to something like "makestep 1.0 3"

[21:55:50] <kl_eisbaer> tampakrap: ...and IMHO "disable monitor" is also missing in the salt profile, but I'm not sure here

[21:56:11] <tampakrap> I'll check

[21:56:27] <kl_eisbaer> cboltz: I've already some other quick and easy things to add to salt

[21:57:02] <tampakrap> disable monitor is there

[21:57:02] <kl_eisbaer> but my last merge requests are long ago - so I probably forgot how to do it properly

[21:57:10] <kl_eisbaer> tampakrap: ah, sorry

[21:57:18] <tampakrap> tinker panic 0 is not

[21:57:22] <tampakrap> add it to all machines?

[21:57:33] <kl_eisbaer> tampakrap: I guess the tinker panic 0 was left out because the initial formula did not support it

[21:57:40] <kl_eisbaer> but all virtual machines should have it

[21:57:52] <tampakrap> only virtual? no physical?

[21:58:03] <kl_eisbaer> because it allows the clock to be stepped

[21:58:04] <tampakrap> ntp servers as well?

[21:58:24] <kl_eisbaer> yes: for virtual machines this might be important during live migration or when they are paused

[21:58:46] <tampakrap> ack

[21:58:59] <kl_eisbaer> some docs also mention that the virtual machines might have problems right after boot, when their "virtual hw clock" is not synced with the hypervisor

[21:59:13] <kl_eisbaer> so it's definitively a setting you want to have for virtual machines

[22:00:07] <pjessen> will have to ask you about that chrony feature tomorrow.

[22:00:11] <kl_eisbaer> the funny part with the "restrict" settings: you can skip all of them with chrony, if you do not plan to use the VM as time server

[22:00:35] <pjessen> ah, better defaults?

[22:00:35] <kl_eisbaer> pjessen: https://chrony.tuxfamily.org/

[22:00:44] <kl_eisbaer> pjessen: more secure defaults, yes

[22:02:00] <kl_eisbaer> but I would say: 2 hours is enough for a meeting ;-)

[22:02:20] <pjessen> yeah, I have to go. Goot meeting though.

[22:02:23] <pjessen> good

[22:02:31] <kl_eisbaer> pjessen: CU

[22:03:06] <pjessen> see ya all.

[22:03:07] <kl_eisbaer> ...and bye, bye to everyone else :-)

[22:03:09] <pjessen> bye

[22:03:20] <Ada_Lovelace> bye

[22:03:21] <cboltz> bye

[22:03:44] <tampakrap> yeah let's finish it