openSUSE admin

2017-07-04 #opensuse-admin - Heroes meeting

[20:02:56] <cboltz> is everybody ready for the meeting?

[20:03:07] <cboltz> my clock says you should be, so let's start ;-)

[20:03:16] <Ada_Lovelace> yes

[20:03:55] <cboltz> does someone from the community have questions?

[20:04:46] <Ada_Lovelace> Seems: no

[20:05:07] <cboltz> ok, so let me continue with the wiki update

[20:05:11] <Ada_Lovelace> Then: wiki update

[20:05:47] <cboltz> Max fetched the latest files and database dump of the english wiki last week

[20:06:10] <cboltz> Theo did the VM setup (+ several "smaller" things like creating databases, nameserver settings etc.)

[20:06:29] <cboltz> the result is that en-test.opensuse.org is now running on Rieslingschorle

[20:06:51] <cboltz> (that is, the riesling VM with Apache, PHP, Mediawiki - and the water VM with elasticsearch)

[20:07:06] <thomic> :D

[20:07:15] <cboltz> so we have an up-to-date test wiki now

[20:07:26] <cboltz> and it is running on the VMs that will go into production

[20:07:32] <cboltz> (of course completely salted ;-)

[20:07:43] <cboltz> my plan is:

[20:08:01] <cboltz> - today: send out another call for testing

[20:08:31] <cboltz> - next week: migrate and update the english wiki (unless testing brings up a showstopper)

[20:08:56] <cboltz> - another week later: migrate and update the other wikis

[20:09:38] <cboltz> I'm quite sure testing will mostly happen _after_ updating the english wiki - this is why I don't want to update all wikis at once ;-)

[20:09:41] <Ada_Lovelace> You read today that we shouldn't do a lot with the openSUSE infrastructure during next weeks (until release)?

[20:09:59] <cboltz> yes, I did ;-)

[20:10:25] <cboltz> we'll see how far we get with updating the non-english wikis

[20:10:47] <Ada_Lovelace> Yes. :_)

[20:10:52] <cboltz> but I really want to have the english wiki updated before the 42.3 release

[20:11:11] <cboltz> if something breaks, we still have enough time before the release to fix it ;-)

[20:11:58] <cboltz> and from the marketing perspective, having the main wiki already running on 42.3 sounds like something we want to have ;-)

[20:12:28] <cboltz> "look how much we trust our new release" ;-)

[20:13:47] <cboltz> BTW: with "next week", I hope for the beginning of the week, which is 2 weeks before the release

[20:14:23] <tampakrap> fine by me

[20:14:40] <mmaher_home> sounds like a solid plan

[20:14:53] <cboltz> :-)

[20:14:59] <Ada_Lovelace> By me, too

[20:15:10] <cboltz> how many days before the release should we stop to migrate non-english wikis?

[20:16:09] <cboltz> (and: should we make that dependent on the size of the wiki?)

[20:16:40] <Ada_Lovelace> I would say 3 or 4 days

[20:17:15] <tampakrap> I would say ask the release managers and the marketing guys

[20:18:09] <Ada_Lovelace> That's a good option

[20:18:38] <cboltz> I'll write a mail to the heroes ML about it anyway, so Ludwig should see it. CC'ing Doug is easy ;-)

[20:19:57] <cboltz> BTW: if the update works without problems (it did for en-test), it means that we have to make the wiki read-only between a) packaging all files and getting a database dump and b) switching over the en.o.o DNS entry

[20:20:33] <cboltz> so a few hours

[20:20:45] <tampakrap> which means you need to announce a downtime

[20:21:06] <cboltz> "just" a read-only time, but yes

[20:21:35] <tampakrap> what do you mean?

[20:21:57] <cboltz> during the update, you can still _read_ the wiki (it will still be served by the Provo server)

[20:22:12] <tampakrap> ah

[20:22:12] <cboltz> but we have to make it read-only before making the database dump

[20:22:17] <tampakrap> the dns change will not happen instantly

[20:22:25] <tampakrap> so still announce it

[20:22:37] <tampakrap> and since it is a big change and it happens before the release

[20:22:44] <Ada_Lovelace> That should be a message on oensuse-project

[20:22:46] <tampakrap> I would say to involve doug to write a news.o.o article

[20:23:11] <tampakrap> it should be on news.o.o, on our blog (before AND after) and on opensuse-announce

[20:23:27] <tampakrap> with a reply-to opensuse-project

[20:24:08] <cboltz> I had planned for a news.o.o article _after_ updating the english wiki

[20:24:36] <cboltz> but I get your point - since we are close to the release, it makes sense to tell people in advance when they can relax instead of updating the wiki ;-)

[20:24:43] <tampakrap> no, it needs one before, and edit that page when it is finished

[20:25:02] <Ada_Lovelace> They would ask on the wiki list and opensuse-project, why they can have only read-only mode at this moment.

[20:25:39] <cboltz> there will be a message in the wiki while it is read-only of course ;-)

[20:25:54] <tampakrap> sure, someone will ask even if you put big giant banners in front of everyone's face

[20:26:10] <cboltz> (something like the "This is a test wiki" we have in en-test.o.o, but of course with a different text)

[20:27:36] <cboltz> for the news.o.o article -

[20:27:44] <cboltz> - should I ask Doug to handle it (which will be interesting for the "done now" update which will happen in the evening)?

[20:27:58] <cboltz> - does one of us (Lars?) have access to news.o.o and can do it or

[20:27:59] <tampakrap> prepare a draft and send it to him

[20:28:06] <cboltz> - should I ask for a news.o.o account?

[20:28:18] <tampakrap> there is a mailing list

[20:28:22] <tampakrap> news@o.o I think

[20:28:28] <tampakrap> prepare a draft and send it there

[20:28:48] <cboltz> ok

[20:29:05] <tampakrap> even if you have your own account there, one extra eyeball pair wouldn't hurt to review

[20:29:24] <cboltz> indeed ;-)

[20:30:31] <cboltz> I think that's it for the wiki update

[20:30:43] <cboltz> we'll see if the testing brings up something

[20:31:05] <cboltz> if everything works without complaints, I'll coordinate the update day/time with tampakrap and mmaher_home

[20:31:21] <cboltz> (probably beginning of next week)

[20:32:03] <cboltz> tampakrap: a last question - can you reduce the TTL of the en.o.o DNS entry?

[20:32:35] <tampakrap> how much is it now?

[20:32:56] <tampakrap> and how much you want it?

[20:33:17] <cboltz> current value is 86400

[20:34:31] <cboltz> my usual TTL for migration is 5 minutes ;-)

[20:34:36] <cboltz> but maybe we should use an hour so that we don't get the load with a "big bang" and have at least a few minutes to provide additional resources

[20:36:27] <tampakrap> I'll ask MF-IT if it is possible

[20:36:31] <tampakrap> but I think it isn't

[20:37:31] <cboltz> correct me if I'm wrong, but AFAIK the DNS zone gets managed in FreeIPA and nsprv*.novell.com are DNS slaves

[20:37:41] <cboltz> so - shouldn't it be possible to set the TTL in FreeIPA?

[20:38:47] <tampakrap> I don't know if we transfer the TTL value as well to them

[20:39:05] <thomic> maybe ask darix, as i know, he was talking about that topic today with lars^^

[20:39:41] <cboltz> ok

[20:40:12] <cboltz> so - anything else for the wiki update?

[20:41:12] <cboltz> doesn''t look so ;-)

[20:41:22] <cboltz> next topic - status reports

[20:41:46] <cboltz> what happens outside the wiki? ;-)

[20:42:54] <Ada_Lovelace> Stanislav upgraded Weblate

[20:43:16] <tampakrap> l10n moved to new VM, now running postgres and with updated version

[20:43:30] <Ada_Lovelace> So it is.

[20:43:45] <tampakrap> that doesn't show statistics on the front page to do a bunch of sql queries by saying good morning and look like a slow webpage finally

[20:44:12] <tampakrap> the migration from mysql to postgres was totally pointless in my opinion, but oh well, he did the job not me

[20:44:31] <tampakrap> also I did some stuff to prepare per's mailman machine

[20:44:38] <tampakrap> and max did some mirror tickets

[20:44:41] <tampakrap> that's all I think

[20:45:08] <mmaher_home> yeah, i got today to the point where i covered every mirror ticket where a costumer is involved so green

[20:45:26] <Ada_Lovelace> He told me the reason that most SUSE databases are running on postgres now

[20:45:42] <tampakrap> who told you that?

[20:45:47] <Ada_Lovelace> Stanislav

[20:46:05] <tampakrap> nope

[20:46:26] <tampakrap> the reason he decided that is because back then the mysql cluster had some issues that caused downtime

[20:47:23] <Ada_Lovelace> https://lists.opensuse.org/opensuse-translation/2017-06/msg00007.html

[20:47:53] <Ada_Lovelace> #

[20:47:54] <Ada_Lovelace> Also openSUSE infrastructure people prefer postgres, as it already

[20:47:56] <Ada_Lovelace> serves large part of the infrastructure.

[20:48:28] <Ada_Lovelace> But unimportant now...

[20:48:48] <tampakrap> I do prefer postgres personally, but when something is already running and a long migration effort is involved, then you balance your choices usually

[20:48:54] <tampakrap> anyway, whatever

[20:49:43] <tampakrap> the migration is done, looks good, congrats to him and thanks for all of his efforts

[20:50:01] <Ada_Lovelace> Yes

[20:51:25] <cboltz> does someone have more status reports, or should we go to the next topic?

[20:52:17] <cboltz> ok, next topic then

[20:52:27] <cboltz> handling of missing tickets in the future

[20:52:41] <cboltz> that might also translate to "handling of spam tickets"

[20:53:23] <cboltz> with the amount of spam we get, it was only a question of time that a ticket gets accidently deleted

[20:53:36] <cboltz> the options I see are:

[20:53:42] <Ada_Lovelace> Do we want to think about a new ticketsystem?

[20:54:07] <cboltz> - continue the current workflow, with the risk of accidently deleting a non-spam ticket

[20:54:47] <cboltz> - close (instead of delete) spam tickets, with the disadvantage that this will send out a mail to the (probably faked) sender address, and the advantage that we could disable the "delete" feature

[20:55:41] <tampakrap> this is not an option

[20:55:54] <cboltz> - "fix" redmine (lots of people will be happy about this!)

[20:55:55] <mmaher_home> close is no option that works for me

[20:55:58] <tampakrap> it may end up in a bounce war

[20:56:06] <cboltz> - switch to another ticket system

[20:56:59] <tampakrap> and migrate content to the new ticketing system or not?

[20:56:59] <cboltz> the perfect solution would obviously be not to receive spam tickets, but I'm afraid reality disagrees ;-)

[20:57:06] <tampakrap> because migration will be almost impossible

[20:57:42] <mmaher_home> can't we install at least a spam filter to reduce the spam tickets?

[20:58:19] <Ada_Lovelace> - using a script for automated restoring of missing tickets via dumps

[20:59:11] <cboltz> we have an offer from Heinlein to give all openSUSE Members a mailbox.org mailbox

[20:59:34] <cboltz> this includes a spam filter (actually blocker) for the other @opensuse.org addresses like admin@

[20:59:37] <mmaher_home> cboltz: yeah this solution would also make the spam situation a lot better. but... whats the status about it?

[21:00:11] <cboltz> it's waiting for the "cleanup" of inactive members

[21:00:46] <cboltz> I don't remember the exact grace time we gave them to respond with an "I'm still active" mail

[21:01:07] <mmaher_home> how long will it take to make the cleanup? just that i have a idea

[21:01:13] <pjessen> am back.

[21:01:13] <cboltz> IIRC it should be done in one or two months

[21:01:52] <thomic> wait? what?

[21:01:56] <thomic> 20:59:10< cboltz> we have an offer from Heinlein to give all openSUSE Members a mailbox.org mailbox

[21:01:59] <thomic> ???

[21:02:13] <mmaher_home> two months of looking a little bit deeper in spam tickets before killing them should be ok for us then. we can endure this ;)

[21:02:36] <pjessen> careful with a spam filter for everyone@o.o it has to be very configurable.

[21:02:42] <thomic> https://media.ccc.de/v/gpn16-7582-run_your_own_fucking_infrastructure <!

[21:02:55] <thomic> what is this channel about?

[21:03:03] <thomic> about moving infrastructure to "the cloud"

[21:03:08] <thomic> ok

[21:03:11] <thomic> .... -.-

[21:03:49] <tampakrap> what is heinlein?

[21:04:02] <thomic> tampakrap: heinlein is the provider of mailbox.org

[21:04:04] <cboltz> thomic: I get your point, but it is a big improvement over the current forward-only mail address we offer

[21:04:05] <thomic> an IT provider

[21:04:10] <Ada_Lovelace> Our sponsor heinlein

[21:04:11] <cboltz> https://heinlein-support.de/

[21:04:20] <thomic> yay one of our sponsors

[21:04:31] <thomic> fine if it's about money or hardware

[21:04:39] <thomic> but their admins control our mailboxes?

[21:04:42] <thomic> hrm

[21:04:47] <pjessen> for admin@o.o I think a challenge-respnse process would be the easiest

[21:05:08] <cboltz> thomic: if I had to choose _one_ company in germany to do mail, it would probably be heinlein ;-)

[21:05:19] <thomic> yay

[21:05:20] <thomic> goodie

[21:05:22] <cboltz> (and I say that as someone who runs several mailservers ;-)

[21:05:24] <Ada_Lovelace> yes

[21:05:40] <thomic> i also can provide other providers which are cool

[21:05:42] <tampakrap> cboltz: if I had to choose _one_ company in germany to do opensuse.org mail, it would definitely be suse

[21:05:48] <thomic> :D

[21:06:01] <pjessen> agree with theo

[21:06:12] <mmaher_home> the thing with heinlein was decided by the board right?

[21:06:22] <tampakrap> mails are sensitive, I don't want them out

[21:06:29] <pjessen> decided??

[21:06:31] <cboltz> tampakrap: I see your point, but reality shows that suse doesn't have a working spamfilter ;-)

[21:07:01] <thomic> wait wot? cboltz?

[21:07:06] <thomic> i have a suse mailaccount

[21:07:12] <thomic> which has a really nice spamfilter

[21:07:13] <thomic> :D

[21:07:24] <thomic> maybe we should setup something similar for o.o

[21:07:32] <thomic> instead of complaining about stuff :)

[21:08:08] <cboltz> opensuse.org mails go through mx2.suse.de

[21:08:15] <thomic> however - if it is a boards' decision - boardmembers should now raise hands

[21:08:17] <cboltz> so I'd expect that the spamfilter also works for admin@o.o

[21:08:43] <thomic> cboltz: mx is the first step - before mail reaches my mailbox - there is a lot more behind

[21:09:09] <cboltz> we heard about heinlein at the admin meeting last December, so before Sarah and I were elected to the board

[21:09:18] <thomic> ah k

[21:09:19] <thomic> so

[21:09:21] <cboltz> nevertheless, I can bring up your concerns at the next board meeting

[21:09:24] <thomic> please clarify this

[21:09:30] <thomic> with rich & others

[21:09:43] <thomic> if boards wants to "outsource" mail services - i'm fine

[21:10:00] <thomic> but than .. heinlein SUPPORT should as well take the ticket queue for this ;)

[21:10:16] <thomic> because - nothing sucks more than playing proxy-admin to an external company

[21:10:18] <mmaher_home> yes please. because i also got the information when i started on the opensuse topic that i have not to worry about spam b.c we go to heinlein. i also know that richard knows about that

[21:10:21] <Ada_Lovelace> of course

[21:10:30] <cboltz> ;-)

[21:10:46] <thomic> i mean its fine

[21:10:54] <thomic> but maybe somebody should talk about that

[21:10:59] <thomic> in case of DNS foo

[21:11:05] <thomic> in case of - how to setup forwardings

[21:11:09] <thomic> in case of migration

[21:11:11] <thomic> etc.

[21:11:24] <thomic> and maybe somebody should inform the community and talk to lawyers

[21:11:38] <thomic> if we (as opensuse project) have licenses and agreements with our users

[21:11:57] <thomic> we need -by german law- an extra agreement if we give out userdata to 3rd parties

[21:12:00] <thomic> like heinlein

[21:12:10] <thomic> has board ever taken considerations like that? :D

[21:12:21] <thomic> it takes month to get this clarified correctly

[21:12:48] * cboltz has a feeling that forwarding this meeting log to the board ML is better than just talking about it in the next meeting

[21:13:08] <thomic> and now i don't start talking about - what is with SUSE employees which are as well community member - does there is there an agreement by work council needed when we have a new 3rd party provider - and a security check?

[21:13:13] <thomic> :D

[21:13:25] <thomic> cboltz: feel free

[21:13:33] <tampakrap> I started a conversation with the board regarding an updated privacy policy to cover the mailing lists and the future mail

[21:13:36] <thomic> I don't want to be the "no-go" man

[21:13:41] <tampakrap> and I have no idea about the status

[21:13:58] <thomic> but there needs to be proper preparation to get such a big change handled by the organization

[21:14:33] <cboltz> thomic: you brought up valid concerns, nobody will be mad at you ;-)

[21:14:49] <cboltz> (actually it's a good thing to discuss this now instead of "too late")

[21:15:13] <thomic> *happy* to work in a bug-report friendly community :) - ah .. reminds me i need to write another bug report ... *please go on with meeting*

[21:15:20] <mmaher_home> so since there is so much other stuff on the topic: spam for now -> still manually

[21:15:58] <thomic> i would say - for now ... keep the workflow going .. but consider to read (even 'spam') a bit more careful before deleting

[21:16:08] <thomic> second - clarify mail situation with board

[21:16:16] <tampakrap> yes but restoring deleted tickets is something I would like to have as Ada_Lovelace proposed

[21:16:32] <thomic> third - if mail is decided - maybe A) problem solves by heinlein-Awesome-Filterness B) think about further steps

[21:16:49] <mmaher_home> if a tickets get deleted and we are aware of it, we can get it back because the mail is still in the admin.o.o inbox. right?

[21:16:55] <thomic> tampakrap: i would like to see this fixed in upstream before scripting on restore-scripts

[21:17:08] <tampakrap> upstream redmine is slow

[21:17:13] <thomic> mmaher_home: not the full conversation inside the ticket - thats the problem

[21:17:14] <tampakrap> we need to take actions here

[21:17:23] <cboltz> thomic: the upstream feature request is about 10 years old IIRC...

[21:17:33] <mmaher_home> thomic: just the initial one. thats enough?

[21:17:44] <thomic> cboltz: yup, i posted it first on the ML :D

[21:18:00] <thomic> mmaher_home: the initial one is not enough

[21:18:06] <thomic> think about following situation

[21:18:11] <Ada_Lovelace> There is a perl script for ticket restoring in the feature request.

[21:18:44] <thomic> what about somebody opening tickets on the page

[21:18:53] <thomic> or an admin does modification or comments

[21:18:58] <thomic> and then it gets deleted

[21:19:41] <cboltz> IMHO the best solution is to subscribe to all tickets (there's an option on the "my account" page to set mail notifications "for all events in all my projects")

[21:19:51] <cboltz> you'll get quite some mails, so filter them into a folder

[21:20:09] <cboltz> if a ticket gets accidently deleted, this mail folder is your personal backup ;-)

[21:20:23] <mmaher_home> not a bad idea

[21:21:26] <cboltz> I have that subscription since December, just in case you need a backup of an older ticket ;-)

[21:21:27] <thomic> ok .. is an idea

[21:23:34] <cboltz> so, to sum it up:

[21:23:56] <cboltz> - we'll keep our workflow and are more careful before/when deleting a spam ticket

[21:24:13] <cboltz> - I'll bring up the mail concerns to the board

[21:25:26] <cboltz> anything else on this topic?

[21:25:30] <tampakrap> yes and if someone wants to play on a deletion recovery system/script, feel free

[21:25:34] <tampakrap> I'd love to have it

[21:26:12] <cboltz> agreed

[21:26:22] <mmaher_home> +

[21:26:25] <Ada_Lovelace> agreed

[21:26:32] <thomic> +1

[21:27:03] <cboltz> having a "real" undelete in redmine would be even better, but that's probably a bigger change - otherwise someone would already have done it ;-)

[21:27:15] <cboltz> (but still, if someone is looking for his/her next hackweek project... ;-)

[21:28:13] <Ada_Lovelace> The base is available...

[21:30:38] <cboltz> any other topic?

[21:30:53] <Ada_Lovelace> http://www.redmine.org/issues/1380#note-24

[21:31:20] <Ada_Lovelace> That should be all for today...

[21:31:51] <pjessen> gotta go. I won't make the next weeting, only just back from Greece and 1 August is our national holiday. Will likely be busy drinking beer ....

[21:32:44] <cboltz> how sad... ;-)

[21:33:18] <cboltz> thanks to everybody who joined the meeting, and also thanks to everybody who helped and will help with the wiki update!