openSUSE admin

2018-07-03 #opensuse-admin - heroes meeting

[19:56:48] <tampakrap> hello

[19:56:53] <pjessen> hi

[19:56:56] <tampakrap> 3 minutes to go!

[19:57:15] <pjessen> goo timing ...

[19:58:23] <pjessen> lets see who else turns up.

[19:58:39] * cboltz hides

[19:59:50] <tampakrap> let's start?

[19:59:58] <pjessen> yup.

[20:00:11] <cboltz> yes

[20:00:23] <tampakrap> cboltz: put your hat on!

[20:00:36] <cboltz> ok, so I'll do a formal

[20:00:40] <cboltz> welcome to the heroes meeting!

[20:00:54] <cboltz> our topics are on https://progress.opensuse.org/issues/36817

[20:01:12] <cboltz> let's start with Q&A from the community - does someone have a question?

[20:01:49] <pjessen> any community members attending?

[20:02:38] <cboltz> looks like nobody has a question ;-)

[20:02:47] <cboltz> so let's continue with status reports

[20:03:18] <cboltz> provo-mirror has some free disk space again - using --delete instead of --delete-after obviously helps

[20:03:59] <tampakrap> I've been experimenting with caasp and cloudfoundry, it seems to work so far but I need to figure out more on how it works

[20:04:19] <tampakrap> the progress is documented here https://trello.com/b/5rcSlDX1/geekops

[20:04:35] <tampakrap> it's also my hackweek project https://hackweek.suse.com/17/projects/microservices-and-serverless-for-the-opensuse-dot-org-infrastructure

[20:04:47] <tampakrap> I intend to spend the last two days on documentation, assuming that it works as expected

[20:04:47] <pjessen> I had to tidy up widehat, get rid of 42.1 stuff. It's still almost full.

[20:06:01] <tampakrap> cboltz: any progress with the sync_update.sh on provo-mirror?

[20:06:05] <pjessen> mirroring - still minor bits that dont work on pontifex.

[20:06:45] <cboltz> you mean why rsync doesn't offer some of the update directories? I didn't have time to check that yet

[20:06:56] <tampakrap> okay

[20:07:15] <tampakrap> pjessen: what doesn't work on pontifex?

[20:07:55] <pjessen> there are some jobs that regularly fail, I've opened at least one ticket. + hotstuff rsnyc packages

[20:08:36] <pjessen> generally the mirroing works okay.

[20:09:24] <tampakrap> cool

[20:09:48] <tampakrap> cboltz: let's move to the next topic?

[20:10:01] <pjessen> I've decided to basically open tickets for whatever stuff I see. Feel free to take a look :-)

[20:10:09] <cboltz> actually I have another status report:

[20:10:17] <cboltz> I just build a Leap 15 JeOS image locally with a minimal amount of packages (not tested yet, but it's probably close to the final version)

[20:10:40] <pjessen> good stuff

[20:10:41] <cboltz> so we'll have an image to deploy Leap 15 machines soon :-)

[20:11:15] <tampakrap> we have a topic about leap15 later

[20:11:32] <pjessen> old tickets? I have not had time to review any.

[20:12:17] <cboltz> same for me, but anyway, I'd say let's move the old tickets to the end of the meeting

[20:12:24] <tampakrap> me neither, I focused on kubernetes the past weeks

[20:12:31] <cboltz> which means the next topic is

[20:12:39] <cboltz> why are we redirecting websites to https?

[20:12:46] <pjessen> right. my question.

[20:13:15] <pjessen> I was wondering why we are forcing everyone to https when we could just have both http and https available? e.g. for lists.o.o

[20:13:51] <tampakrap> is there a reason to not have https everywhere?

[20:14:24] <pjessen> I had a discussion about using older browsers which are okay with https, but not with TLS 1.2

[20:14:53] <pjessen> by forcing https and tls1.2 we are excluding some people. dunno how many.

[20:15:33] <pjessen> tls1.2 is fine, but I think it might be good to offer both http and https. when the content isnt sensitve.

[20:15:58] <cboltz> well, define "sensitive" ;-)

[20:16:14] <pjessen> well, not sensitive = lists.o.o for instance.

[20:16:15] <cboltz> if the NSA catches you reading lots of mails about GPG and disk encryption - is this sensitive or not?

[20:16:37] <tampakrap> since we didn't get any reports about this yet, I would like to change it

[20:16:38] <pjessen> if the NSA is after me, uising http is the least of my worries

[20:17:00] <cboltz> ;-)

[20:17:08] <cboltz> https also ensures that nobody can do a MITM and inject a "rm -rf /" in some example script

[20:17:18] <cboltz> (or, closer to the real world, inject ads)

[20:18:23] <pjessen> for insensitive data, I would like us to leave it to the user to choose.

[20:18:35] <tampakrap> sorry, I *wouldn't like to change it

[20:18:48] <pjessen> yeah, I thought thats what you meant. :-)

[20:19:17] <cboltz> I also think the redirect to https makes sense

[20:19:20] <pjessen> it was mentioned briefly on opensuse@o.o, that's why I added the topic.

[20:19:54] <cboltz> right, I followed that discussion ;-)

[20:20:00] <tampakrap> is there a link?

[20:20:20] <pjessen> not right now. I can dig it out

[20:20:51] <cboltz> tampakrap: let me warn you that it's a typical opensuse@ discussion, so you'll need some time to read all the mails ;-)

[20:21:03] <tampakrap> okay

[20:21:26] <tampakrap> I'll read and let you know if I am convinced, but for now I am not

[20:22:04] <cboltz> another detail that was mentioned in this discussion is that www.o.o, bugzilla and forums (basically the things running in provo) don't support the latest tls

[20:22:15] <cboltz> do you think there's a chance to get this fixed/updated?

[20:22:28] <pjessen> okay. I would say why not suppoert both.

[20:22:40] <tampakrap> we can file the ticket to MF-IT and see what happens

[20:22:58] <pjessen> how long will MF-IT be looking after things?

[20:23:11] <tampakrap> cboltz: file me a ticket to progress.o.o and I'll forward it to them so I don't forget

[20:23:29] <cboltz> ok

[20:23:51] <tampakrap> pjessen: can't answer that, it usually depends on the urgency of the ticket

[20:24:31] <tampakrap> anything else or can we move to the next one?

[20:24:47] <pjessen> lets move on

[20:25:10] <cboltz> ok, next topic is the planned provo downtime on July 13th

[20:25:29] <tampakrap> yes, also mentioned to https://progress.opensuse.org/issues/37543

[20:25:47] <pjessen> link to https discussion - https://lists.opensuse.org/opensuse/2018-06/msg00752.html

[20:26:01] <tampakrap> anybody willing to create a draft for our status.o.o and opensuse-announce@o.o announcements?

[20:26:29] <bmwiedemann> hi there. had to get kids to bed.

[20:26:59] <tampakrap> hello bmwiedemann

[20:27:07] <pjessen> hi bernd

[20:27:25] <bmwiedemann> could we re-use some of the last announcement of the scheduled downtime?

[20:27:31] <tampakrap> sure

[20:27:56] <bmwiedemann> shall we start a page on etherpad.opensuse.org ?

[20:28:11] <tampakrap> sure good idea

[20:28:43] <tampakrap> I would say to send the announcement on opensuse-announce@o.o with a link to the status.o.o incident this week asap

[20:28:59] <tampakrap> then one reminder three days before, and one more reminder the same day

[20:29:08] <tampakrap> sounds good?

[20:29:18] <pjessen> yep, thats how it got to be done

[20:29:25] <cboltz> yes

[20:29:29] <bmwiedemann> https://etherpad.opensuse.org/p/adminoo-downtime20180713

[20:30:07] <cboltz> the downtime is expected to last two hours, so we don't need to setup replacements for most of the services

[20:30:10] <tampakrap> bmwiedemann: would you like to start the draft?

[20:30:18] <cboltz> the only thing that might be worth some effort is www.o.o

[20:30:35] <cboltz> that's a static page, so temporarily moving it to NBG shouldn't be too hard

[20:30:53] <cboltz> (yes, I'm ignoring the openID endpoint ;-)

[20:31:32] <tampakrap> agreed, any volunteers? :)

[20:31:36] <tampakrap> it could be done on static.o.o

[20:31:42] <pjessen> the announcement has to list which services are affected.

[20:33:10] <tampakrap> so somebody already started the draft on etherpad (put your name on the top right corner please)

[20:33:45] <tampakrap> and since nobody volunteered, I'll make sure we get the announcement on status.o.o and on the opensuse-announce ml

[20:33:54] <tampakrap> and I'll work on the www.o.o temp replacement

[20:34:25] <tampakrap> anything else?

[20:34:26] <cboltz> bmwiedemann: seeing your questionmark on the pad - the ticket on progress lists the affected services

[20:34:49] <cboltz> IIRC our public nameservers also run in provo

[20:34:55] <cboltz> will they be affected by the outage?

[20:35:57] <pjessen> looks like there is also one in houston - nshou1.novell.com. ?

[20:36:35] <bmwiedemann> have a link to the ticket? the tool is not very good at finding these things

[20:36:48] <tampakrap> I will ask about this, but I think MF-IT maintains nameservers not only in provo

[20:36:53] <cboltz> bmwiedemann: https://progress.opensuse.org/issues/37543

[20:37:48] <cboltz> tampakrap: also ask if we can deploy DNS updates during the outage, or if we have to do it in advance

[20:38:24] <cboltz> if we have to switch over www.o.o in advance, it might be a good idea to ProxyPass (or however it's called in haproxy) /openid/ to the server in provo

[20:38:37] <tampakrap> okay

[20:40:02] <cboltz> next topic?

[20:40:41] <cboltz> since nobody objects ;-) let's continue with

[20:40:43] <cboltz> leap 15 status

[20:41:13] <cboltz> I already mentioned that I'm working on the JeOS image

[20:41:19] <tampakrap> yey!! good job!

[20:41:28] <cboltz> I didn't test the latest version yet, but I expect it to work ;-)

[20:41:43] <tampakrap> we have also machines running leap15 already

[20:41:47] <tampakrap> one is gitlab.i.o.o

[20:41:56] <bmwiedemann> cboltz: we got an OpenStack image in OBS, so if you drop cloud-init there, you are close to JeOS

[20:42:16] <tampakrap> the others are the new postgresql cluster machines, mirrordb1 and mirrordb2

[20:42:35] <tampakrap> darix managed to figure out a performance issue we were having, so the cluster is ready for use

[20:42:53] <tampakrap> and we can start moving databases from the old cluster mirrordb[3-4] already

[20:43:03] <tampakrap> I'll probably start working on it after hackweek

[20:43:23] <cboltz> tampakrap: did I miss the salt code to setup those machines? ;-)

[20:43:23] <tampakrap> so something important that is missing is leap15 repos in salt

[20:44:01] <tampakrap> exactly! you need to start with the repos!

[20:44:28] <cboltz> I'd hope that we can basically cat 42.sls 12_or_42.sls > 15.sls

[20:44:48] <cboltz> (not the best solution, we'll have to clean that up after dropping SLE 11)

[20:44:52] <tampakrap> maybe, I'll leave that to you

[20:45:15] <tampakrap> the good thing is that we don't need sle15 repos, as we don't plan to have any sle15 machines

[20:46:10] <cboltz> regarding Leap 15 repos - I'm quite sure we have them in salt already ;-)

[20:46:31] <cboltz> IIRC we have a leap.sls that says something like download.o.o/..../$version/

[20:46:45] <cboltz> which should work for Leap 15 without changes

[20:46:57] <tampakrap> mind giving it a try?

[20:47:03] <tampakrap> I didn't have time to check it yet

[20:47:16] <cboltz> I will, when testing my shiny new JeOS image ;-)

[20:48:08] <tampakrap> perfect

[20:48:22] <tampakrap> do you have any ETA for that?

[20:49:04] <cboltz> this or next week, depends on which other things get onto my TODO list

[20:49:11] <tampakrap> cool

[20:49:37] <tampakrap> anything else apart from the jeos and the salt code for the repos needed for leap15 before we start migrating VMs?

[20:50:51] <cboltz> in theory, it should just work ;-)

[20:50:59] <pjessen> famous last words

[20:51:14] <tampakrap> now I'm curious

[20:51:15] <cboltz> in practise, I'd prefer to upgrade some of my test VMs before upgrading the production machines ;-)

[20:51:27] <tampakrap> I'll do a salt-call state.apply test=True on mickey

[20:51:50] <tampakrap> mickey had to be upgraded, because there was a conflict between gitlab and sssd packages

[20:51:58] <tampakrap> that was preventing me from updating gitlab

[20:52:05] <tampakrap> mickey (gitlab.i.o.o):~ # salt-call state.apply test=True

[20:52:07] <tampakrap> local:

[20:52:09] <tampakrap> Data failed to compile:

[20:52:11] <tampakrap> ----------

[20:52:13] <tampakrap> Pillar failed to render with the following messages:

[20:52:15] <tampakrap> ----------

[20:52:16] <cboltz> you'll probably get an error for osmajorrelease/15 pillar

[20:52:17] <tampakrap> Specified SLS 'osmajorrelease.15' in environment 'production' is not available on the salt master

[20:52:19] <tampakrap> so, it needs work :)

[20:52:22] <bmwiedemann> is that using salt-master/minion? because salt-ssh has some compat issues with python2 vs 3

[20:53:04] <cboltz> tampakrap: looks as expected ;-) - but fixing that should be easy

[20:53:25] <tampakrap> bmwiedemann: master/minion, we don't use salt-ssh

[20:54:45] <tampakrap> anything else or can we move?

[20:55:14] <cboltz> IMHO we can continue with the next topic

[20:55:19] <cboltz> extra headers on download.o.o

[20:55:40] <tampakrap> yes that's mine

[20:55:53] <pjessen> lets hear it

[20:56:16] <tampakrap> lars forwarded me a mail a couple of weeks ago

[20:56:37] <tampakrap> a guy from microfocus is running some security scans on opensuse.org services

[20:56:48] <tampakrap> and he requested to add some extra headers to download.o.o

[20:57:24] <pjessen> interesting

[20:57:28] <tampakrap> Cache-Control,Content-Security-Policy,Strict-Transport-Security,X-Content-Type-Options

[20:57:41] <tampakrap> I am copy pasting from a csv, I might be pasting wrong content

[20:57:45] <tampakrap> give me a few secs

[20:57:52] <pjessen> cache-control, no prob

[20:58:07] <pjessen> content-security-policy- dunno

[20:58:17] <bmwiedemann> Strict mode might be overkill.

[20:58:19] <pjessen> STS - questionable. Maybe.

[20:59:19] <pjessen> I think STS would be okay, but a bit pointless. Most requests are redirected to http anyway.

[20:59:25] <tampakrap> Lars says that apart from the Content-Security-Policy cookie, the rest seem fine to him

[20:59:25] <bmwiedemann> it would prevent users from browsing the non-https version

[20:59:55] <pjessen> they are already prevented I think. See earlier topic from tonight :-)

[21:00:17] <tampakrap> download.o.o is the only one afair that runs on http as well

[21:00:42] <pjessen> yes, my mistake, just checked it

[21:00:46] <tampakrap> https://progress.opensuse.org/news/40 Lars wrote this article 8 months ago, still valid

[21:03:19] <pjessen> which precludes HSTS

[21:03:44] <tampakrap> STS I think is already there

[21:04:31] <bmwiedemann> not on download.o.o

[21:05:18] <bmwiedemann> says curl -I

[21:05:26] <pjessen> says my browser too

[21:05:35] <tampakrap> ah maybe the tool doesn't really need it

[21:05:47] <tampakrap> No security headers are set - cache control, content security policy, x content type options are missing.

[21:05:54] <tampakrap> that's the summary from the report

[21:06:21] <pjessen> the only I am familiar with is cache control and hsts. Never heard of the others

[21:07:13] <tampakrap> me neither, I'm googling about x-content-type-options

[21:07:14] <bmwiedemann> the content* stuff might be relevant to HTML/JS stuff which we dont host there

[21:07:41] <tampakrap> https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options

[21:08:37] <bmwiedemann> yes. XSS and data-injection are not really relevant there.

[21:08:44] <tampakrap> so I would suggest to enable cache-control and x-content-type-options, announce it and see what happens

[21:08:55] <tampakrap> just to make the colleague's tool happy :)

[21:09:06] <pjessen> what would he like for cache-control ?

[21:10:00] <tampakrap> it is not defined, I'll ask him

[21:10:20] <pjessen> its more of a header for .. wait for it ..... cache control

[21:11:06] <tampakrap> I suppose the tool doesn't care much, as long as it is set

[21:11:06] <bmwiedemann> like telling proxies and browsers which files they can re-use later

[21:11:16] <pjessen> exactly

[21:11:18] <tampakrap> so better to ask if he has a recommended value

[21:11:40] <pjessen> the thing is - its not very security related.

[21:11:54] <pjessen> ask him and see what he says.

[21:12:00] <tampakrap> okay

[21:12:08] <bmwiedemann> it could, if shared proxies could give out private data or such

[21:12:30] <cboltz> given how often the content of download.o.o changes (basically every minute), don't allow too much caching ;-)

[21:12:33] <bmwiedemann> (but that should be avoided with a "Vary" header anyway)

[21:12:43] <pjessen> yep.

[21:12:51] <pjessen> well, every 5 mins?

[21:13:18] <bmwiedemann> can be useful to be able to use the 'back' button without having a forced page reload

[21:13:52] <pjessen> I think it wud be sensible.

[21:14:35] <pjessen> I'll take a closer look at those options. is there an open ticket already?

[21:15:23] <tampakrap> no, can you create one please?

[21:15:27] <pjessen> willdo

[21:15:32] <tampakrap> thanks

[21:15:51] <bmwiedemann> slightly off-topic: I sometimes find that "Short overview" confusing, because it is above what I'm actually interested in

[21:16:22] <tampakrap> so to sum up: cache-control AI for me to ask about expected values, content-security-policy we don't want, x-content-type-options no problem to do

[21:16:24] <tampakrap> correct?

[21:16:49] <pjessen> I dont know what content-sec-policy does yet

[21:17:01] <pjessen> hsts wwe don't want

[21:18:01] <cboltz> pjessen: https://media.ccc.de/v/gpn18-141-http-security-headers might be helpful (if you have an hour to watch it)

[21:18:23] <pjessen> I dont :-)

[21:19:16] <tampakrap> bmwiedemann: which short overview?

[21:21:54] <bmwiedemann> tampakrap: on each download.o.o page, the HTML has the same links in a list

[21:22:12] <bmwiedemann> that is rather similar to directory listings I'm usually looking for

[21:22:43] <tampakrap> ah yes

[21:23:35] <tampakrap> I assume it comes from mirrorbrain

[21:24:08] <pjessen> no, I think itŝ generated by some or other script, based on a template

[21:24:18] <cboltz> tampakrap: I doubt - IIRC it's a simple template for the directory index

[21:24:27] <bmwiedemann> at least it is not on the mirrors.

[21:24:50] <pjessen> yes, the mirrors do their own thing.

[21:25:41] <bmwiedemann> one quick fix could be moving it from header to footer

[21:25:52] <bmwiedemann> or reducing font size / contrast

[21:26:20] <tampakrap> btw, can we close the meeting or is there anything else to discuss?

[21:26:54] <pjessen> im done. ticket opened: https://progress.opensuse.org/issues/38159

[21:27:40] <cboltz> I'd say it's late enough to skip checking old tickets ;-)

[21:30:06] <pjessen> okay, then I'll say goodbye. I might not be back for the August meeting. I think I'm travelling.

[21:32:18] <cboltz> found the header and footer - HEADER.html and README.html in /usr/share/apache2/mirrorbrain-icons

[21:32:36] <cboltz> both files are packaged in mirrorbrain-icons, so if you want to change them, please do it in the package ;-)

[21:33:45] <cboltz> that said - I also think we can close the meeting

[21:33:52] <cboltz> thanks everybody!