openSUSE admin

2019-07-02 heroes meeting

[19:58:59] <knurpht> hey all

[19:59:34] <knurpht> hey Per

[19:59:43] <pjessen> hi gertjan

[19:59:48] <tuanpembual> hi all

[19:59:57] <knurpht> tuanpembual: hi

[20:00:03] <cboltz> hi!

[20:00:23] <knurpht> cboltz: you were in 'all' :P

[20:00:33] <cboltz> ;-)

[20:01:01] <cboltz> let's start the meeting ;-)

[20:01:15] <cboltz> our usual topics are on https://progress.opensuse.org/issues/52565 - but we can still add whatever is needed

[20:01:34] <cboltz> does someone from the community have questions?

[20:01:48] <knurpht> yep

[20:02:08] <knurpht> #topic: the stuff from Provo, what is the status ?

[20:03:34] <cboltz> I didn't see any answers (besides the one asking for the used OS and browser when the login problems happen, which I answered with a "most probably openSUSE, and firefox or chromium as browser")

[20:04:01] <cboltz> and since then: silence

[20:04:06] <knurpht> fyi chrome has these issues too.

[20:04:33] <cboltz> I'd be surprised if it's a browser issue - server problems are more likely

[20:04:41] <cboltz> okurz: IIRC you opened some tickets, did you get a more useful answer?

[20:05:07] <knurpht> But, is anything going on moving the services to N'berg / wherever instead of Provo?

[20:05:25] <okurz[m]> Unfortunately none

[20:05:47] <pjessen> carlos has done some work on getting a new news-server running, I don't have any update from him.

[20:07:20] <pjessen> the current config cannot just be copied over, it is not inn.

[20:08:10] <pjessen> I'm not sure how much cooperation he has from provo wrt to getting information on the bi-dir interface

[20:08:41] <pjessen> maybe it is just slow because of the time of year

[20:09:12] <pjessen> ah, there he is.

[20:09:17] <knurpht> hi carlos

[20:09:42] <robin_listas> I'm on the road with a laptop. Hi

[20:09:55] <pjessen> welcome!

[20:12:39] <pjessen> carlos, any update on progress with the news-server?

[20:13:44] <robin_listas> Well, the only thing I can do is study at home.

[20:14:15] <knurpht> Does the board have to escalate this to SUSE?

[20:15:05] <bmwiedemann> good evening

[20:15:28] <robin_listas> I don't know, I'm new on this. A news server needs a machine and volunteers. Volunteers there are some.

[20:15:57] <knurpht> bmwiedemann: hi

[20:16:49] <pjessen> afaict, provo/MF-IT is not a show-stopper just yet?

[20:17:26] <cboltz> robin_listas: knurpht's question is probably about things like getting information how the news server can connect to the forums

[20:17:53] <robin_listas> Ah, ok

[20:18:22] <cboltz> so, to start with - did someone already ask the admins in Provo for this information?

[20:18:25] <knurpht> IIRC there's an issue since the NNTP service was hired somewhere.

[20:18:46] <cboltz> (ideally ask in a ticket so that there's something to escalate if needed)

[20:18:57] <knurpht> cboltz: Kim Groneman can be contacted.

[20:19:27] <cboltz> knurpht: we'll need to know how our nntp server can connect to the forums

[20:19:34] <knurpht> but he no longer has access to our services.

[20:19:48] <cboltz> great :-(

[20:19:58] <pjessen> my guess - via email, but it would be useful to understand the current setup.

[20:22:05] <knurpht> #question: do we want all the work and than later this year switch to an alternative for the VB forums?

[20:22:19] <knurpht> f.e. Discourse?

[20:22:41] <knurpht> that seems to be more 'open'.

[20:23:15] <cboltz> the answer probably depends on how much work it is to setup a nntp server for the current forums now

[20:23:34] <knurpht> cboltz: agreed

[20:23:35] <robin_listas> Part of the work I suppose would keep. The gateway, dunno

[20:23:38] <pjessen> and if we have the access needed to reconfigure the forums.

[20:25:33] <pjessen> knurpht: if Kim G does have the access, who does?

[20:25:47] <pjessen> not have to access

[20:26:25] <knurpht> pjessen: No idea, I forwarded what Kim told me.

[20:26:49] <knurpht> better ask him directly

[20:27:01] <knurpht> I lack the knowledge in this area.

[20:27:52] <pjessen> okay. I have emailed with him, but he seemed a little "reluctant"

[20:29:40] <knurpht> pjessen: that's why I asked about escalating it to SUSE, they're the ones paying MF for services. I would think transferring the stuff should be included.

[20:29:59] <pjessen> right, good point.

[20:30:45] <pjessen> I think we should probably delay that until MF-IT actually becomes a show-stopper?

[20:31:03] <knurpht> Yup

[20:31:06] <cboltz> knurpht: I'm quite sure the first question is "which ticket number", so - as much as I like mails - make sure you have a ticket ;-)

[20:31:20] <pjessen> +1

[20:32:01] <cboltz> given the usual "speed" of MF-IT it might be a good idea to prepare the server setup and ask for the needed config to connect to the forums in parallel

[20:32:29] <pjessen> yes, absolutely.

[20:36:27] <pjessen> knurpht - I am sort of assuming it would be helpful to consider also moving the forums to a macine in Nuernberg too?

[20:36:51] <knurpht> absolutely

[20:37:13] <robin_listas> Should be a single machine for nntp and forum, or two?

[20:37:32] <cboltz> separate machines/VMs please

[20:37:36] <knurpht> AFAIK they're separate now.

[20:37:45] <pjessen> uh - why not just one?

[20:37:50] <bmwiedemann> everything is VMs there these days

[20:38:04] <pjessen> two machines is only more admin.

[20:38:19] <cboltz> it can make things easier in the future

[20:38:23] <pjessen> the newsserver will be tiny.

[20:38:41] <cboltz> for example, when we switch to another forum software, we can simply switch off the VM with the old software

[20:39:01] <cboltz> instead of having to cleanup everything, while carefully not touching the nntp server

[20:39:05] <pjessen> okay, it's no big deal.

[20:39:10] <knurpht> cboltz: yes.

[20:39:11] <bmwiedemann> pjessen: imagine nntp does disk/mem fillup - if it is a separate VM, it will not hurt forums

[20:39:37] <knurpht> bmwiedemann: so a VM per service?

[20:40:05] <robin_listas> Currently, the nntp message base (opensuse.org only) was 4.5 GB only

[20:42:20] <pjessen> knurpht: can you help with export/importing the forum contents to a new forum setup?

[20:43:31] <bmwiedemann> knurpht: it is cleaner at least. and with salt, the admin overhead is not that high

[20:43:46] <knurpht> I might be able to pull some strings here and there, but I alread have a lot on my desk currently.

[20:44:13] <pjessen> "currently" is less important :-) it might not be until August or september

[20:44:53] <pjessen> I was more interested if you knew how to do it personally.

[20:45:11] <knurpht> pjessen: then count on me.

[20:45:17] <pjessen> cool.

[20:45:45] <pjessen> who creates VMs these days?

[20:46:55] <cboltz> mcaj or kbabioch should be able to do that

[20:47:31] <pjessen> I did ask Martin a while back, but he's been very busy.

[20:48:45] <cboltz> all SUSE admins are very busy, but that shouldn't stop you ;-)

[20:49:37] <pjessen> I'll check with mcaj again.

[20:49:46] <cboltz> BTW: Martin created a VM for me last week, killing my excuse to delay the wiki update ;-)

[20:50:19] <pjessen> I'm obviously not being persistent enough.

[20:50:39] <cboltz> when did you ask him?

[20:50:42] <knurpht> pjessen: No, you're not cboltz :P

[20:50:49] <pjessen> not sure, maybe 2-3 weeks ago.

[20:50:57] <cboltz> (IIRC my request took a few months)

[20:50:57] <pjessen> haha

[20:51:43] <pjessen> maybe we should leave this topic for now ?

[20:51:47] <cboltz> 2-3 weeks? Sorry to say that, but you are way behind me in the queue ;-)

[20:51:49] <knurpht> yup

[20:52:15] <knurpht> next #topic ?

[20:52:26] <cboltz> status reports

[20:52:47] <knurpht> yep, who updates status.o.o ?

[20:52:49] <kbabioch> yes, we are indeed very busy :-) ... is there a ticket for this vm that you need?

[20:53:04] <pjessen> there will be soon :-)

[20:53:22] <kbabioch> and feel free to remind us often. this will also create some awareness in management that we need more ressources

[20:53:36] <kbabioch> because most of this work is not appreciated at all, since there is not a lot of noise if things work out ;-)

[20:54:19] <cboltz> right, admins only get noticed if something breaks :-(

[20:55:39] <cboltz> knurpht: several people have accounts to update status.o.o - basically whoever notices an outage or works on it

[20:56:02] <cboltz> that said - do you have/want an account there?

[20:56:24] <knurpht> cboltz: Don't have one, and yes

[20:56:47] <cboltz> ok, you'll get a mail in some minutes

[20:58:25] <cboltz> kbabioch: I guess you also want access to status.o.o?

[20:58:36] <kbabioch> yeah, sure, wouldn't mind ;-)

[20:58:51] <robin_listas> Ok, as you are now on different topic, I'll leave for now. I'm parked somewhere with engine running (for AC). Unless you want me for something?

[20:59:00] <knurpht> kbabioch: nice, we can meet there :P

[20:59:25] <knurpht> robin_listas: we know where to find you ......

[20:59:42] <robin_listas> Ok. Bye!

[21:00:03] <cboltz> kbabioch: you have mail ;-)

[21:00:50] <kbabioch> yup, i have, thank you :-)

[21:01:19] <cboltz> so - who has status updates?

[21:01:32] <cboltz> quick one from me: as you probably noticed on the ML, I finally moved counter.o.o to a Leap 15.1 VM - one service less running on SLE 11 :-)

[21:02:09] <bmwiedemann> how many others are on SLE11?

[21:02:59] <cboltz> I'd have to count because these old vms (especially community.infra.o.o and boosters.infra.o.o) each run multiple services

[21:03:36] <cboltz> and to make things even more interesting, these VMs are of course not documented...

[21:05:25] <cboltz> for web services, grepping the haproxy config is probably the best idea

[21:05:48] <cboltz> but there are probably a few other services "hidden" on them ;-)

[21:07:00] <cboltz> speaking about haproxy - currently the haproxy.cfg on anna/elsa is only readable for root

[21:07:27] <cboltz> does that make sense, or should we make it readable for everybody who has VPN/ssh access?

[21:08:59] <kbabioch> did you actively change it or is this the package default?

[21:09:33] <cboltz> good question, give me a second to check it...

[21:09:49] <bmwiedemann> also, are there any secrets in there?

[21:10:07] <bmwiedemann> in case someone breaks into another service on that machine or such

[21:10:13] <cboltz> root-only seems to be the default mode

[21:10:59] <cboltz> the file basically has a list of service -> IP, like "the wiki runs on 192.168.47.42"

[21:11:37] <cboltz> I don't think that qualifies as top-secret ;-)

[21:15:56] <bmwiedemann> should be fine to be 644 then

[21:16:08] <cboltz> I also think so

[21:17:26] <bmwiedemann> or you could be sophisticated and make it 640 root:users

[21:18:22] <cboltz> our setup would require root:openvpn - we don't use the "users" group ;-)

[21:18:38] <kbabioch> how many users do have vpn access, but are not root?

[21:18:54] <kbabioch> dont know the permission model / details in opensuse heroes yet ;-)

[21:19:24] <bmwiedemann> kbabioch: I for example

[21:19:28] <cboltz> most people have root/sudo on specific VMs

[21:19:54] <cboltz> but not too many have sudo permissions on all VMs (members of the "wheel" group)

[21:20:00] <mstroeder> Being the user management guy I have to admit that root:openvpn looks pretty awkward to me...

[21:20:49] * Pharaoh_Atem waves

[21:21:04] <cboltz> I know, but it seems to be the default group, and some people are only in that group

[21:21:13] <Pharaoh_Atem> cboltz, mstroeder: I'm pretty close to getting basic openSUSE specs going for ipsilon and lasso

[21:21:21] <cboltz> and it's still better than 644 ;-)

[21:22:02] <kbabioch> well, don't know nginx's permission model, but make sure to restart everything and verify that it still works :-)

[21:22:27] <cboltz> s/nginx/haproxy/ ;-)

[21:22:32] <kbabioch> ah, right

[21:22:47] <cboltz> besides that - done on anna

[21:24:21] <cboltz> I'm not sure if the sync to elsa works, but as soon as someone applies (and checks) the pending keepalived config changes from salt, I'll put the haproxy config into salt ;-)

[21:25:27] <cboltz> (I'm not familiar with keepalived, and therefore won't blindly deploy these changes)

[21:30:39] <pjessen> next topic?

[21:30:49] <pjessen> or status report

[21:31:39] <pjessen> i guess we have widehat back up running again?

[21:34:11] <bmwiedemann> thanks kbabioch :-)

[21:39:06] * bmwiedemann is off to bed

[21:39:26] <cboltz> I know that mcaj did several things during hack week, but it seems he is away

[21:39:41] <cboltz> does someone else have a status report?

[21:42:00] <cboltz> doesn't look so ;-)

[21:42:45] <cboltz> in theory the next topic is "review of old tickets" - that would make lots of sense, but it's (once more) late enough to skip it ;-)

[21:43:16] <cboltz> does someone have another topic, or should we close the meeting?

[21:45:08] <cboltz> I'll take the silence as a vote to close the meeting ;-)

[21:45:25] <cboltz> thank you all for joining!

[21:51:09] <pjessen> sorry, I was busy elsewhere. Good meeting

[21:52:52] <knurpht> sorry interrupted by phonecall

[21:53:30] <pjessen> if anyone cares - it's hot and sticky. even only 23C out side. hope fully the johannisbeeren will be ready for picking in two weeks b4 we go on holidays

[21:54:40] <pjessen> gotta go too, pick up my wife from the train. knurpht, I'll get back to you wrt a new forums setup.

[21:56:19] <cboltz> pjessen: it was also terribly cold here today - only 27 °C after ~40 °C last week ;-)