2018-07-03 #opensuse-admin - heroes meeting [19:56:48] hello [19:56:53] hi [19:56:56] 3 minutes to go! [19:57:15] goo timing ... [19:58:23] lets see who else turns up. [19:58:39] * cboltz hides [19:59:50] let's start? [19:59:58] yup. [20:00:11] yes [20:00:23] cboltz: put your hat on! [20:00:36] ok, so I'll do a formal [20:00:40] welcome to the heroes meeting! [20:00:54] our topics are on https://progress.opensuse.org/issues/36817 [20:01:12] let's start with Q&A from the community - does someone have a question? [20:01:49] any community members attending? [20:02:38] looks like nobody has a question ;-) [20:02:47] so let's continue with status reports [20:03:18] provo-mirror has some free disk space again - using --delete instead of --delete-after obviously helps [20:03:59] I've been experimenting with caasp and cloudfoundry, it seems to work so far but I need to figure out more on how it works [20:04:19] the progress is documented here https://trello.com/b/5rcSlDX1/geekops [20:04:35] it's also my hackweek project https://hackweek.suse.com/17/projects/microservices-and-serverless-for-the-opensuse-dot-org-infrastructure [20:04:47] I intend to spend the last two days on documentation, assuming that it works as expected [20:04:47] I had to tidy up widehat, get rid of 42.1 stuff. It's still almost full. [20:06:01] cboltz: any progress with the sync_update.sh on provo-mirror? [20:06:05] mirroring - still minor bits that dont work on pontifex. [20:06:45] you mean why rsync doesn't offer some of the update directories? I didn't have time to check that yet [20:06:56] okay [20:07:15] pjessen: what doesn't work on pontifex? [20:07:55] there are some jobs that regularly fail, I've opened at least one ticket. + hotstuff rsnyc packages [20:08:36] generally the mirroing works okay. [20:09:24] cool [20:09:48] cboltz: let's move to the next topic? [20:10:01] I've decided to basically open tickets for whatever stuff I see. Feel free to take a look :-) [20:10:09] actually I have another status report: [20:10:17] I just build a Leap 15 JeOS image locally with a minimal amount of packages (not tested yet, but it's probably close to the final version) [20:10:40] good stuff [20:10:41] so we'll have an image to deploy Leap 15 machines soon :-) [20:11:15] we have a topic about leap15 later [20:11:32] old tickets? I have not had time to review any. [20:12:17] same for me, but anyway, I'd say let's move the old tickets to the end of the meeting [20:12:24] me neither, I focused on kubernetes the past weeks [20:12:31] which means the next topic is [20:12:39] why are we redirecting websites to https? [20:12:46] right. my question. [20:13:15] I was wondering why we are forcing everyone to https when we could just have both http and https available? e.g. for lists.o.o [20:13:51] is there a reason to not have https everywhere? [20:14:24] I had a discussion about using older browsers which are okay with https, but not with TLS 1.2 [20:14:53] by forcing https and tls1.2 we are excluding some people. dunno how many. [20:15:33] tls1.2 is fine, but I think it might be good to offer both http and https. when the content isnt sensitve. [20:15:58] well, define "sensitive" ;-) [20:16:14] well, not sensitive = lists.o.o for instance. [20:16:15] if the NSA catches you reading lots of mails about GPG and disk encryption - is this sensitive or not? [20:16:37] since we didn't get any reports about this yet, I would like to change it [20:16:38] if the NSA is after me, uising http is the least of my worries [20:17:00] ;-) [20:17:08] https also ensures that nobody can do a MITM and inject a "rm -rf /" in some example script [20:17:18] (or, closer to the real world, inject ads) [20:18:23] for insensitive data, I would like us to leave it to the user to choose. [20:18:35] sorry, I *wouldn't like to change it [20:18:48] yeah, I thought thats what you meant. :-) [20:19:17] I also think the redirect to https makes sense [20:19:20] it was mentioned briefly on opensuse@o.o, that's why I added the topic. [20:19:54] right, I followed that discussion ;-) [20:20:00] is there a link? [20:20:20] not right now. I can dig it out [20:20:51] tampakrap: let me warn you that it's a typical opensuse@ discussion, so you'll need some time to read all the mails ;-) [20:21:03] okay [20:21:26] I'll read and let you know if I am convinced, but for now I am not [20:22:04] another detail that was mentioned in this discussion is that www.o.o, bugzilla and forums (basically the things running in provo) don't support the latest tls [20:22:15] do you think there's a chance to get this fixed/updated? [20:22:28] okay. I would say why not suppoert both. [20:22:40] we can file the ticket to MF-IT and see what happens [20:22:58] how long will MF-IT be looking after things? [20:23:11] cboltz: file me a ticket to progress.o.o and I'll forward it to them so I don't forget [20:23:29] ok [20:23:51] pjessen: can't answer that, it usually depends on the urgency of the ticket [20:24:31] anything else or can we move to the next one? [20:24:47] lets move on [20:25:10] ok, next topic is the planned provo downtime on July 13th [20:25:29] yes, also mentioned to https://progress.opensuse.org/issues/37543 [20:25:47] link to https discussion - https://lists.opensuse.org/opensuse/2018-06/msg00752.html [20:26:01] anybody willing to create a draft for our status.o.o and opensuse-announce@o.o announcements? [20:26:29] hi there. had to get kids to bed. [20:26:59] hello bmwiedemann [20:27:07] hi bernd [20:27:25] could we re-use some of the last announcement of the scheduled downtime? [20:27:31] sure [20:27:56] shall we start a page on etherpad.opensuse.org ? [20:28:11] sure good idea [20:28:43] I would say to send the announcement on opensuse-announce@o.o with a link to the status.o.o incident this week asap [20:28:59] then one reminder three days before, and one more reminder the same day [20:29:08] sounds good? [20:29:18] yep, thats how it got to be done [20:29:25] yes [20:29:29] https://etherpad.opensuse.org/p/adminoo-downtime20180713 [20:30:07] the downtime is expected to last two hours, so we don't need to setup replacements for most of the services [20:30:10] bmwiedemann: would you like to start the draft? [20:30:18] the only thing that might be worth some effort is www.o.o [20:30:35] that's a static page, so temporarily moving it to NBG shouldn't be too hard [20:30:53] (yes, I'm ignoring the openID endpoint ;-) [20:31:32] agreed, any volunteers? :) [20:31:36] it could be done on static.o.o [20:31:42] the announcement has to list which services are affected. [20:33:10] so somebody already started the draft on etherpad (put your name on the top right corner please) [20:33:45] and since nobody volunteered, I'll make sure we get the announcement on status.o.o and on the opensuse-announce ml [20:33:54] and I'll work on the www.o.o temp replacement [20:34:25] anything else? [20:34:26] bmwiedemann: seeing your questionmark on the pad - the ticket on progress lists the affected services [20:34:49] IIRC our public nameservers also run in provo [20:34:55] will they be affected by the outage? [20:35:57] looks like there is also one in houston - nshou1.novell.com. ? [20:36:35] have a link to the ticket? the tool is not very good at finding these things [20:36:48] I will ask about this, but I think MF-IT maintains nameservers not only in provo [20:36:53] bmwiedemann: https://progress.opensuse.org/issues/37543 [20:37:48] tampakrap: also ask if we can deploy DNS updates during the outage, or if we have to do it in advance [20:38:24] if we have to switch over www.o.o in advance, it might be a good idea to ProxyPass (or however it's called in haproxy) /openid/ to the server in provo [20:38:37] okay [20:40:02] next topic? [20:40:41] since nobody objects ;-) let's continue with [20:40:43] leap 15 status [20:41:13] I already mentioned that I'm working on the JeOS image [20:41:19] yey!! good job! [20:41:28] I didn't test the latest version yet, but I expect it to work ;-) [20:41:43] we have also machines running leap15 already [20:41:47] one is gitlab.i.o.o [20:41:56] cboltz: we got an OpenStack image in OBS, so if you drop cloud-init there, you are close to JeOS [20:42:16] the others are the new postgresql cluster machines, mirrordb1 and mirrordb2 [20:42:35] darix managed to figure out a performance issue we were having, so the cluster is ready for use [20:42:53] and we can start moving databases from the old cluster mirrordb[3-4] already [20:43:03] I'll probably start working on it after hackweek [20:43:23] tampakrap: did I miss the salt code to setup those machines? ;-) [20:43:23] so something important that is missing is leap15 repos in salt [20:44:01] exactly! you need to start with the repos! [20:44:28] I'd hope that we can basically cat 42.sls 12_or_42.sls > 15.sls [20:44:48] (not the best solution, we'll have to clean that up after dropping SLE 11) [20:44:52] maybe, I'll leave that to you [20:45:15] the good thing is that we don't need sle15 repos, as we don't plan to have any sle15 machines [20:46:10] regarding Leap 15 repos - I'm quite sure we have them in salt already ;-) [20:46:31] IIRC we have a leap.sls that says something like download.o.o/..../$version/ [20:46:45] which should work for Leap 15 without changes [20:46:57] mind giving it a try? [20:47:03] I didn't have time to check it yet [20:47:16] I will, when testing my shiny new JeOS image ;-) [20:48:08] perfect [20:48:22] do you have any ETA for that? [20:49:04] this or next week, depends on which other things get onto my TODO list [20:49:11] cool [20:49:37] anything else apart from the jeos and the salt code for the repos needed for leap15 before we start migrating VMs? [20:50:51] in theory, it should just work ;-) [20:50:59] famous last words [20:51:14] now I'm curious [20:51:15] in practise, I'd prefer to upgrade some of my test VMs before upgrading the production machines ;-) [20:51:27] I'll do a salt-call state.apply test=True on mickey [20:51:50] mickey had to be upgraded, because there was a conflict between gitlab and sssd packages [20:51:58] that was preventing me from updating gitlab [20:52:05] mickey (gitlab.i.o.o):~ # salt-call state.apply test=True [20:52:07] local: [20:52:09] Data failed to compile: [20:52:11] ---------- [20:52:13] Pillar failed to render with the following messages: [20:52:15] ---------- [20:52:16] you'll probably get an error for osmajorrelease/15 pillar [20:52:17] Specified SLS 'osmajorrelease.15' in environment 'production' is not available on the salt master [20:52:19] so, it needs work :) [20:52:22] is that using salt-master/minion? because salt-ssh has some compat issues with python2 vs 3 [20:53:04] tampakrap: looks as expected ;-) - but fixing that should be easy [20:53:25] bmwiedemann: master/minion, we don't use salt-ssh [20:54:45] anything else or can we move? [20:55:14] IMHO we can continue with the next topic [20:55:19] extra headers on download.o.o [20:55:40] yes that's mine [20:55:53] lets hear it [20:56:16] lars forwarded me a mail a couple of weeks ago [20:56:37] a guy from microfocus is running some security scans on opensuse.org services [20:56:48] and he requested to add some extra headers to download.o.o [20:57:24] interesting [20:57:28] Cache-Control,Content-Security-Policy,Strict-Transport-Security,X-Content-Type-Options [20:57:41] I am copy pasting from a csv, I might be pasting wrong content [20:57:45] give me a few secs [20:57:52] cache-control, no prob [20:58:07] content-security-policy- dunno [20:58:17] Strict mode might be overkill. [20:58:19] STS - questionable. Maybe. [20:59:19] I think STS would be okay, but a bit pointless. Most requests are redirected to http anyway. [20:59:25] Lars says that apart from the Content-Security-Policy cookie, the rest seem fine to him [20:59:25] it would prevent users from browsing the non-https version [20:59:55] they are already prevented I think. See earlier topic from tonight :-) [21:00:17] download.o.o is the only one afair that runs on http as well [21:00:42] yes, my mistake, just checked it [21:00:46] https://progress.opensuse.org/news/40 Lars wrote this article 8 months ago, still valid [21:03:19] which precludes HSTS [21:03:44] STS I think is already there [21:04:31] not on download.o.o [21:05:18] says curl -I [21:05:26] says my browser too [21:05:35] ah maybe the tool doesn't really need it [21:05:47] No security headers are set - cache control, content security policy, x content type options are missing. [21:05:54] that's the summary from the report [21:06:21] the only I am familiar with is cache control and hsts. Never heard of the others [21:07:13] me neither, I'm googling about x-content-type-options [21:07:14] the content* stuff might be relevant to HTML/JS stuff which we dont host there [21:07:41] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options [21:08:37] yes. XSS and data-injection are not really relevant there. [21:08:44] so I would suggest to enable cache-control and x-content-type-options, announce it and see what happens [21:08:55] just to make the colleague's tool happy :) [21:09:06] what would he like for cache-control ? [21:10:00] it is not defined, I'll ask him [21:10:20] its more of a header for .. wait for it ..... cache control [21:11:06] I suppose the tool doesn't care much, as long as it is set [21:11:06] like telling proxies and browsers which files they can re-use later [21:11:16] exactly [21:11:18] so better to ask if he has a recommended value [21:11:40] the thing is - its not very security related. [21:11:54] ask him and see what he says. [21:12:00] okay [21:12:08] it could, if shared proxies could give out private data or such [21:12:30] given how often the content of download.o.o changes (basically every minute), don't allow too much caching ;-) [21:12:33] (but that should be avoided with a "Vary" header anyway) [21:12:43] yep. [21:12:51] well, every 5 mins? [21:13:18] can be useful to be able to use the 'back' button without having a forced page reload [21:13:52] I think it wud be sensible. [21:14:35] I'll take a closer look at those options. is there an open ticket already? [21:15:23] no, can you create one please? [21:15:27] willdo [21:15:32] thanks [21:15:51] slightly off-topic: I sometimes find that "Short overview" confusing, because it is above what I'm actually interested in [21:16:22] so to sum up: cache-control AI for me to ask about expected values, content-security-policy we don't want, x-content-type-options no problem to do [21:16:24] correct? [21:16:49] I dont know what content-sec-policy does yet [21:17:01] hsts wwe don't want [21:18:01] pjessen: https://media.ccc.de/v/gpn18-141-http-security-headers might be helpful (if you have an hour to watch it) [21:18:23] I dont :-) [21:19:16] bmwiedemann: which short overview? [21:21:54] tampakrap: on each download.o.o page, the HTML has the same links in a list [21:22:12] that is rather similar to directory listings I'm usually looking for [21:22:43] ah yes [21:23:35] I assume it comes from mirrorbrain [21:24:08] no, I think itŝ generated by some or other script, based on a template [21:24:18] tampakrap: I doubt - IIRC it's a simple template for the directory index [21:24:27] at least it is not on the mirrors. [21:24:50] yes, the mirrors do their own thing. [21:25:41] one quick fix could be moving it from header to footer [21:25:52] or reducing font size / contrast [21:26:20] btw, can we close the meeting or is there anything else to discuss? [21:26:54] im done. ticket opened: https://progress.opensuse.org/issues/38159 [21:27:40] I'd say it's late enough to skip checking old tickets ;-) [21:30:06] okay, then I'll say goodbye. I might not be back for the August meeting. I think I'm travelling. [21:32:18] found the header and footer - HEADER.html and README.html in /usr/share/apache2/mirrorbrain-icons [21:32:36] both files are packaged in mirrorbrain-icons, so if you want to change them, please do it in the package ;-) [21:33:45] that said - I also think we can close the meeting [21:33:52] thanks everybody!