2019-05-07 #opensuse-admin meeting

[20:02:32] <cboltz> is someone here for the heroes meeting?
[20:27:40] <bmwiedemann> Hi
[20:27:53] <cboltz> hi
[20:28:04] <bmwiedemann> did I miss the heroes meeting?
[20:28:18] <cboltz> not really, nobody showed up
[20:28:25] <mcaj> HI I'm also a bit late
[20:28:50] <bmwiedemann> Hi Martin
[20:28:57] <cboltz> hi
[20:29:01] <mcaj> cboltz: you are not alone  ;)
[20:29:07] <cboltz> :-)
[20:29:27] <mcaj> and I hope we will see each other on osc 19
[20:29:38] <cboltz> yes, I'll be there
[20:29:44] <Fraser_Bell> cboltz:  Yeah, but *I* am alone.
[20:29:51] <mcaj> I think there is a lot to talk about it
[20:30:16] <bmwiedemann> I will also be there at least Fri and Sat
[20:31:01] <cboltz> Michael will give a talk about Æ-DIR on sunday, please don't miss it ;-)
[20:31:14] <mcaj> For example the future of connect.opensuse.org. That machine needs be turn off soon .
[20:31:27] <mcaj> yep I will be there.
[20:31:52] <cboltz> (IMHO Æ-DIR it could replace FreeIPA,but let's talk about that after Michael's talk)
[20:32:42] <cboltz> regarding connect - fully agreed, I hope we have a replacement ready soon
[20:32:45] <bmwiedemann> ae-DIR shows as ?-DIR here in pidgin
[20:33:18] <cboltz> looks like pidgin has utf-8 issues, or you just need to switch to a bette font
[20:33:51] <bmwiedemann> may be a font issue. did not use that VM much otherwise.
[20:34:41] <cboltz> one thing I added to today's agenda is
[20:34:42] <Fraser_Bell> Same here in HexChat
[20:34:46] <cboltz>     move content of https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Machines pages to pillar/id 
[20:35:05] <mcaj> cboltz: its more the politic desition the  technical...
[20:35:21] <mcaj> let look there
[20:35:25] <Fraser_Bell> yet, from bmwiedemann, it shows properly
[20:35:33] <cboltz> the reason why I propose this is that pillar/id/* is maintained, while the wiki pages are, well, less maintained ;-)
[20:35:40] <bmwiedemann> Fraser_Bell: because I wrote it as ascii
[20:35:45] <Fraser_Bell> as ae-DIR
[20:35:57] <Fraser_Bell> ?Did that show?
[20:36:01] <bmwiedemann> yes
[20:36:17] <cboltz> mcaj: so, what's your polictical ;-) opinion about moving the machine details to pillar/id/ ?
[20:36:19] <Fraser_Bell> Try it without ASCII
[20:36:42] <bmwiedemann> cboltz: would it be gpg-encrypted then?
[20:36:44] <Fraser_Bell> If it works, we narrow it down to cboltz
[20:37:23] <cboltz> bmwiedemann: no, and I see no reason to encrypt it ;-)
[20:37:35] <cboltz> for example, look at https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Annainfraopensuseorg
[20:37:45] <cboltz> is there anything top-secret in it? I doubt ;-)
[20:38:04] <bmwiedemann> was just wondering because I saw some cert renew touched an encrypted blob, though certs are public
[20:38:18] <Fraser_Bell> Yes.  the a & the e is top secret, apparently.
[20:38:58] <cboltz> ;-)
[20:39:06] <mcaj> cboltz: I have mo problem to do it. But If we have a machine there we do a kernel update and machine is not able boot any more, then I see a problem...
[20:39:33] <cboltz> bmwiedemann: we have some things gpg-encrypted, for example database passwords
[20:39:48] <cboltz> and there's also a separate pass repo with root passwords etc.
[20:40:44] <cboltz> mcaj: moving the machine info to pillar/id/ might even help in that case because (I hope) you have cloned the salt repo locally
[20:41:15] <cboltz> hi pjessen 
[20:41:17] <pjessen> sorry for being late, something went wrong with konversation
[20:41:31] <bmwiedemann> cboltz: distributed information is indeed less likely to get lost
[20:41:48] <cboltz> pjessen: don't worry, we started only 5 minutes ago because most people were late
[20:41:49] <mcaj> well sure let do it , BTW what is the procedure, create a MR for it ? (fine with me) but who do the code review ?
[20:42:18] <pjessen> haha, thanks
[20:42:19] <mcaj> pjessen: keine stress, we just start ;)
[20:42:32] <cboltz> yes, create a MR
[20:42:42] <cboltz> the review can be done by each of us
[20:43:32] <mcaj> I need to check the email setting there ... I think i'm not getting emails from infra gitlab ...
[20:43:52] * mcaj knows how to do it
[20:44:33] <cboltz> my experience is that you should simply subscribe to "everything from the salt repo"
[20:44:43] <cboltz> (no worries, you won't get a mail flood ;-)
[20:45:20] <mcaj> BTW what about the machine rsync.opensuse.org  I spoke with Tbro and looks like this machine has some problems with stability ...
[20:45:56] <cboltz> I don't know all details
[20:46:00] <cboltz> what I know is:
[20:46:15] <cboltz> - the machine was replaced after the old one broke last year
[20:46:20] <pjessen> right.
[20:46:24] <mcaj> yes
[20:46:24] <cboltz> - the replacement is a used OBS server
[20:46:39] <cboltz> - the harddisks are too slow for the traffic the machine gets
[20:46:45] <mcaj> so it s a bare metal machine ?
[20:47:07] <cboltz> yes, hosted at a sponsor's datacenter
[20:47:13] <bmwiedemann> there are several public mirrors that provide rsync, so I use these instead. Maybe we could convince one of those operators to let us point rsync.o.o there?
[20:47:45] * mcaj have an idea : start donation for the server and buy new fast disks for it ^^ :)
[20:48:07] <pjessen> i think rsync.o.o might be superfluous
[20:48:50] <pjessen> anyone who wants a private mirror can create one with wget, from the rest of the mirrors
[20:49:19] <bmwiedemann> wget sucks for that. but rsync from public mirrors is fine.
[20:50:14] <pjessen> basically rsync.o.o is at best a "nice to have", that's my piont.
[20:50:18] <cboltz> there's also the idea to rent some https://www.hetzner.de/dedicated-rootserver/sx62 servers to run rsync.o.o
[20:51:16] <bmwiedemann> that one has spinning rust though, so only 100 IOPS
[20:51:31] <mcaj> yes but I think  the core server should be the best what we can have...  why to run in on 5 or more old SATA disks ...
[20:51:34] <pjessen> last I checked the usage (I don't have access to widehat right now), I counted some 10 mirrors I think.
[20:52:13] <pjessen> max thruput is 100MB/s anyway. (on a 1Gbps link
[20:52:28] <pjessen> 10 users, not 10 mirrors
[20:52:46] <cboltz> bmwiedemann: I know, but for the disk space we need, renting 4 of them is probably still cheaper than one server with enough SSDs
[20:52:48] <mcaj> pjessen: Why you do not have it ? I think specially you should have it ...
[20:53:07] <pjessen> i wonder if tbro hasn't had time to set it up.
[20:53:54] <mcaj> pjessen: I will ping him in two days.  We need to fix this!
[20:54:08] <pjessen> okay, sounds good.
[20:55:25] <mcaj> check ;)
[20:55:48] <cboltz> mcaj: do you have some time to setup two VMs for me?
[20:56:08] <mcaj> maybe ^^ :)
[20:56:10] * Fraser_Bell agrees with mcaj re. donations?
[20:56:33] <cboltz> ok, so my wishlist ;-) is:
[20:56:49] <mcaj> cboltz: send me the mail/request with spec about them.
[20:57:06] <cboltz> ok
[20:57:14] <mcaj> tomorrow i a day off but by end of the wee I can do it.
[20:58:06] <cboltz> one detail I'll already mention here is that these VMs should run 15.1 ;-)
[20:58:19] <cboltz> (I created/updated the JeOS image some weeks ago)
[20:59:05] <Fraser_Bell> welcome to the meeting, knurpht
[20:59:09] <mcaj> cboltz: NP to me :d
[20:59:58] <knurpht> cboltz: Could you maybe start with your 'Couple of things' ?
[21:00:05] <cboltz> as a sidenote - cboltz.de and blog.cboltz.de is already running on 15.1 since yesterday
[21:00:14] <cboltz> ;-)
[21:00:53] <knurpht> Meh. *.knurpht.nl is running Tumbleweed :-)
[21:00:57] <pjessen> yeah for servers, 15.1 is looking very good.
[21:01:23] <Fraser_Bell> pjessen, et al:  THAT is nice to hear!
[21:01:30] <cboltz> I'd love to do that, but there are more domains on that server, and they don't like daily upgrades too much
[21:01:54] <knurpht> cboltz: plan is to move the thing to 15.1
[21:02:10] <mcaj> I test TW for servers and welll leap is leap
[21:02:23] <cboltz> (especially new PHP versions can be interesting[tm] - but it was a good reason to finally upgrade some things ;-)
[21:02:39] <Fraser_Bell> mcaj:  Actually, leap is Leap
[21:02:52] <mcaj> works but need constant maintanance with some m. windows ...
[21:03:04] <mcaj> ja ja ja
[21:03:14] <pjessen> where are we on the meeting agenda?
[21:03:24] <Fraser_Bell> ^ ??
[21:03:30] <mcaj> PHP is nightmare ..
[21:03:38] <cboltz> pjessen: we use /dev/random as meeting agenda pointer today ;-)
[21:04:06] <Fraser_Bell> cboltz: ROTFL
[21:04:07] <pjessen> ah, sorry. I guess I missed the introduction :-)
[21:04:11] <mcaj> the main point should be : book a hotel and visit OSC 19 !
[21:04:38] <cboltz> exactly
[21:04:39] <Fraser_Bell> Too late, mcaj:  Maybe oSC 20?
[21:04:56] <mcaj> and there let talk about it. ATM the infra is fine but I see some week points there ...
[21:05:14] <Fraser_Bell> week? not daily?
[21:05:18] <knurpht> mcaj: Be quick, Doug has warned lots of times on all platforms, that there is some huge opticians congress the same weekend.
[21:05:27] <cboltz> as long as you don't see _weak_ points... ;-)
[21:05:48] <pjessen> I wont make it, got other priorities.
[21:06:04] <mcaj> knurpht: I can not comment that since I'm in SUSE...
[21:06:07] <mcaj> but
[21:06:20] <mcaj> that Su** Fu** ...
[21:06:24] <mcaj> :(
[21:06:46] <knurpht> SUSE has tents in Fürst.
[21:07:06] <mcaj> And Ask there who made that time frame ...
[21:07:15] <pjessen> it's May, termperatures only go slightly below
[21:08:02] <bmwiedemann> we had 2 C some morning this week
[21:08:03] * knurpht slept a night outside at oSC17 due to hotel room being > 40°C
[21:08:31] <pjessen> ice on the car this morning.
[21:08:54] <cboltz> knurpht: sounds like "free sauna" ;-)
[21:08:59] <tuanpembual> Hi
[21:09:08] <cboltz> hi
[21:09:12] <tuanpembual> sorry being late,
[21:09:23] <tuanpembual> wrong timezone agaie :(
[21:09:31] <Fraser_Bell> We is hitting 30C/89F here in Kamloops in a day or two
[21:10:16] <bmwiedemann> Fraser_Bell: in Canada?
[21:10:43] <Fraser_Bell> Of course.  Where else, since I do not have a passport?
[21:10:56] <mcaj> tuanpembual: are you the correct person who is working ATM on redmine upgrade ?
[21:11:20] <knurpht> Canada replaced passports with grass
[21:11:28] <tuanpembual> yes mcaj
[21:11:37] <Fraser_Bell> Actually, the whole country went to pot.
[21:11:44] <tuanpembual> but not finish yet.
[21:11:51] <Fraser_Bell> & US has been Trumped
[21:12:27] <mcaj> tuanpembual: I know but is the any progress do you need any help from ENG INFRA team or from the Heores team ?
[21:12:31] <Fraser_Bell> tuanpembual:  Thanks for working on it.
[21:13:27] <tuanpembual> mcaj: I cannot login to database cluster from the server
[21:13:36] <tuanpembual> connect server
[21:13:48] <mcaj> ok
[21:14:22] <tuanpembual> tampakrap dont write password on crendential file.
[21:14:30] <mcaj> tuanpembual: can tyou send me your GPG key ? I will collect and send you need passwords ...
[21:15:03] <tuanpembual> noted. will dm you my gpg
[21:15:28] <mcaj> ack
[21:17:52] <Fraser_Bell> I think we need to look really hard at getting our infrastructure on our own servers under our own control, should start whatever campaign we need to get sponsors and get that done, as there are far too many location, connection, and similar issues the way things are.  f.e.:  (and only one of many) I tried adding the 15.1 release counter on my servers in Florida, New York, and Eastern Canada.  Somehow it is blocked
[21:17:53] <Fraser_Bell> (certificate? or?) and will not show up.
[21:18:37] <knurpht> Fraser_Bell: see latest Board meeting minutes.
[21:18:47] <knurpht> on project ML
[21:18:48] <Fraser_Bell> ... and, as a webmaster, along with all else I am currently working on, I have no time to troubleshoot.
[21:19:06] <cboltz> Fraser_Bell: the counter is hosted on our own servers (well, SUSE servers running openSUSE VMs) so we have full access
[21:19:25] <pjessen>  open a ticket?
[21:19:25] <cboltz> if you tell me where embedding fails, I can check it
[21:19:35] <Fraser_Bell> SUSE servers, and most of my connection problems I run into are SUSE-based.
[21:19:46] <Fraser_Bell> Including accessing the Forums, from time to time.
[21:20:01] <bmwiedemann> we had some IPv6 trouble in some places
[21:20:39] <cboltz> the forums live in Provo on Microfocus servers, which is a totally different story ;-)
[21:21:09] <Fraser_Bell> Yes, but waiting for suse.com and waiting for microfocus is about equal.
[21:21:33] <cboltz> the funny thing is that suse.com is also hosted by microfocus
[21:21:59] <mcaj> Thanks all, Im looking forward to see you in a few weeks in Z-Bau and I need to go now..
[21:22:14] <bmwiedemann> mcaj: have a good time.
[21:22:16] <Fraser_Bell> mcaj: ciau
[21:22:22] <knurpht> mcaj: see you there
[21:22:38] <pjessen> thanks mcaj - same here, gotta go. (unless anyone's got anything important for me?)
[21:22:52] <mcaj> I'm always here but in "invisible" mode :d
[21:23:14] <Fraser_Bell> LOL I feel that way sometimes, too.
[21:24:18] <Fraser_Bell> Leaving.  See you next time, maybe.
[21:35:08] <bmwiedemann> good night
[21:35:27] <cboltz> good night
[22:05:28] <tuanpembual> good night all.
[22:14:53] <knurpht> good night