Project

General

Profile

communication #56435 ยป 2019-10-01-heroes-meeting.txt

IRC meeting log - cboltz, 2019-10-01 22:19

 
2019-10-01 #opensuse-admin heroes meeting
[20:00:57] <cboltz> Hi everybody!
[20:01:03] <cboltz> who is here for the heroes meeting?
[20:01:57] <mstroeder> michael@ae-dir.com
[20:02:44] <klein> rklein@suse.com
[20:02:59] <cboltz> hi!
[20:03:11] <cboltz> (no need to show up by mail address ;-)
[20:03:17] <tuanpembual> hi
[20:03:39] <tuanpembual> me for heroes meeting
[20:03:51] * klein newbie, don't know the "protocol"
[20:04:33] <cboltz> well, it's a meeting ;-)
[20:04:45] <cboltz> officially we have a schedule, see https://progress.opensuse.org/issues/56435
[20:05:13] <cboltz> but in practise we don't handle it that strict and just discuss what needs to be discussed ;-)
[20:05:24] <cboltz> the only somewhat fixed item is:
[20:05:32] <cboltz> does someone from the community have a question?
[20:06:16] <cboltz> (that often results in silence, but I'm happy about everybody who surprises me/us, so please ask ;-)
[20:06:41] <pjessen> I'm back
[20:06:46] <tuanpembual> hi pjessen
[20:06:49] <tuanpembual> :D
[20:06:55] <cboltz> hi!
[20:07:35] <cboltz> looks like we can continue with the next topic - status reports about everything
[20:07:40] <klein> do we have a current diagram of our infra and a list of "mission critical systems" that need to be running?
[20:07:59] * kbabioch is also here
[20:08:20] <cboltz> not exactly
[20:08:37] <cboltz> we have the monitoring, and we also have the list of machines in salt (pillar/id/*)
[20:09:25] <cboltz> and maybe a few people have a list of mission critical systems in /dev/brain ;-) - but nothing written down AFAIK
[20:09:39] <klein> ok, I will see the pillars and then ask more questions in the next meeting.
[20:10:21] <cboltz> you can also ask earlier (here or on the mailinglist), no need to wait a month ;-)
[20:10:37] <klein> ok
[20:11:15] <mstroeder> Missing documentation make it really hard for volunteers to fix something on existing systems. Diagrams can be easily coded with http://blockdiag.com for systems and networks.
[20:11:52] <cboltz> I fully agree that missing documentation makes it hard
[20:12:13] <cboltz> but I slightly doubt that we need diagrams - our setup isn't that complicated IMHO ;-)
[20:12:17] <klein> but too much docs makes it harder too :-)
[20:12:54] <cboltz> (typically $service = haproxy + actual (web) server + maybe database server)
[20:13:10] <mstroeder> There is much room between no docs and too many docs.
[20:13:24] <cboltz> I've rarely seen "too many docs" ;-)
[20:14:36] <cboltz> since we are already discussing this:
[20:14:52] <cboltz> we have https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Machines (most likely outdated)
[20:14:58] <cboltz> and we have some data in salt pillar/id/*
[20:15:13] <mstroeder> Without blaming somebody I'd like to point out the Let's Encrypt fixes raised on the mailing list during the last days. The custom stuff and its dependencies are too hard to find out while in a hurry.
[20:15:52] <cboltz> no objections ;-)
[20:16:17] <kbabioch> mstroeder: yup, its heavily customized (hook script for dehydrated) ... but to be fair: not sure how else to do it, since we're using freeipa for dns management and want to have wildcard certs (i.e. we need dns authentication)
[20:17:10] <kbabioch> and it took 2 persons 1,5 hours or so, so afterall its not too bad, since we also had to "hack" our way into the machine (no credentials were documented ;-))
[20:17:12] <klein> I dont see dehydated as a bad thing... it has a post/pre hook feature, we just need to document it better
[20:18:31] <kbabioch> let's put it this way (and this is a serious quesiton): has anyone ever worked in an environment where documentation was good/sufficient? because everyone always wants to have it, but in my experience its always a challenge
[20:19:09] <kbabioch> in some places there wasn't any, in some other places there is outdated documentation (which is even worse) ... but never seen a place, where everything was perfect in terms of documentation ...
[20:19:15] <klein> sorry, I mean just have a README in /root telling that you should use dehydrated and kinit, nothing that fancy
[20:19:40] <klein> the rest, one can figure by himself
[20:20:28] <klein> more than that, will make us ending up with outdated docs
[20:21:26] <cboltz> personally I'd prefer to document that with some pkg.installed in salt ;-) (maybe with some comments added) but in general I agree that some basic docs would be useful
[20:21:43] <kbabioch> i agree ... but we're currently even struggling with figuring out if machines are "reboot_safe" ... i have my doubts if we will have time/bandwith to work on those kin of docs
[20:22:08] <pjessen> me too
[20:22:27] <cboltz> well, do it like you do it with reboot_safe - whenever you find out something worth documenting, document it
[20:22:30] <klein> maybe that list of servers and what they do (a short description) would help to know wich ones are safe and the ones that need more love
[20:22:47] <cboltz> even if it's just a comment in a salt file, that's much better than nothing
[20:23:17] <mcaj_away> Hi all
[20:23:20] <klein> yeah, maybe we can find a way to have good pillars or someting that moves with the living infra
[20:23:37] <klein> and not some wiki that no one updates
[20:23:58] <cboltz> can you read my /dev/brain? I just wanted to propose to move the content of the machine pages into pillar/id/ ;-)
[20:24:11] * mcaj_away is on a pub wifi ...
[20:24:29] <pjessen> hi martin
[20:24:30] <cboltz> that's more likely to get updated than wiki pages, and has a smaller risk of getting out of sync
[20:24:37] <klein> cboltz: :-) that makes perfect sense
[20:25:11] <mcaj_away> pjessen: hi, a pub wifi had some packetloss, but now is fine
[20:25:41] <cboltz> give me some days to come up with a proof-of-concept merge request showing it for a few hosts
[20:26:13] <cboltz> I'll send the link to the mailinglist, and if nobody complains, do it for more machines
[20:26:16] <tuanpembual> I dont have access to pillar or gitlab.
[20:26:18] <tuanpembual> :D
[20:26:43] <cboltz> that's something we'll need to fix ;-)
[20:26:47] <tuanpembual> who can I request to this?
[20:27:03] <mcaj_away> we can fix that just send an email to admin@opensuse.org
[20:27:10] <mcaj_away> I can fix it ...
[20:27:43] <tuanpembual> oke mcaj_away, will send email. thanks
[20:28:04] <tuanpembual> I do usually write tec doc everyday.
[20:28:18] <tuanpembual> using markdown at private gitlab.
[20:28:27] <tuanpembual> maybe I can help more.
[20:28:56] <tuanpembual> I want report my progress.
[20:29:41] <cboltz> go ahead ;-)
[20:29:43] <tuanpembual> https://progress.opensuse.org/issues/27720 last status, need pointing domain.
[20:30:22] <tuanpembual> that all.
[20:31:23] <cboltz> so - who can setup progress-test.o.o as CNAME login2.o.o for tuanpembual (ideally _now_ ;-) ?
[20:32:20] * cboltz doesn't have permissions to add DNS entries
[20:32:27] <mcaj_away> tuanpembual: try gitlab your account is not "unblock"
[20:32:40] <mcaj_away> s/not/now
[20:32:55] <tuanpembual> sure.
[20:33:34] <kbabioch> i can do it now
[20:34:46] <tuanpembual> thank mcaj_away, it work.
[20:35:09] <mcaj_away> see this is the admin super power ;)
[20:35:39] <cboltz> mcaj_away: the second half is/was to give tuanpembual developer access to https://gitlab.infra.opensuse.org/infra/salt/ - which I did before tuanpembual tested ;-)
[20:36:06] <kbabioch> progress-test.opensuse.org is setup ... might take a while for the zone transfer to happen
[20:36:13] <cboltz> (AFAIK that repo is only accessible for project members, not for random gitlab users ;-)
[20:36:18] <cboltz> thanks!
[20:36:21] <kbabioch> but actually it already works
[20:37:09] <cboltz> it still needs some config in haproxy
[20:37:10] <tuanpembual> I will continue tuning new redmine.
[20:37:24] <tuanpembual> *pointing domain etc.
[20:37:31] <mcaj_away> done
[20:37:38] <tuanpembual> thank kbabioch
[20:38:25] <mcaj_away> btw did you already speak about helios ? https://progress.opensuse.org/issues/57104
[20:38:39] <cboltz> not yet, but we should ;-)
[20:38:40] <kbabioch> nope, we didn't
[20:39:01] <cboltz> my proposal is:
[20:39:21] <cboltz> - as a quickfix (because we need it next week) downgrade to 42.3
[20:39:40] <cboltz> - longer term, either upgrade everything or find a replacement
[20:40:01] <kbabioch> downgrading will also mean to run an outdated / unsupported machine ... not exactly a good idea :-/
[20:40:33] <mcaj_away> yes we have this SLA we can not have out of support OS
[20:40:38] <cboltz> I know (and hate it as much as everybody else), but not having an election tool when we need it is not better :-/
[20:41:01] <mstroeder> Error message "No module named django" means that Django is not installed.
[20:41:07] <mcaj_away> we should maybe send a request do comunity, developers or board
[20:41:20] <mcaj_away> about fix the tool or find new one
[20:41:59] <mstroeder> Or Django module package is not readable due to missing AppArmor rules...
[20:42:02] <mcaj_away> the problem with it is that is out of date and not compatible and secure with currect python verison
[20:42:32] <cboltz> mstroeder: the funny thing is that Django _is_ installed (at least as far as I can see), and IIRC there isn't an AppArmor profile on it
[20:43:31] <cboltz> which means the error message is IMHO somewhat misleading
[20:43:42] <cboltz> mcaj_away: did you look into more details?
[20:43:58] <mstroeder> Is it a Py2/Py3 compat issue?
[20:44:37] <mcaj_away> I spoke wiht Tomas Chvatal and he told me two years ago the tool was problematic, now days it just not secure and fixible without changes on srource code/ upstream
[20:45:34] <mcaj_away> you can look here https://heliosvoting.org/ and here https://github.com/benadida/helios-server
[20:47:03] * kbabioch thinks that we shouldn't run an election tool if we cannot manage to package, maintain and operate it :-/
[20:47:24] <mcaj_away> and there is no update on upstream
[20:47:27] <cboltz> yeah, but not running an election tool also isn't an option...
[20:48:08] <mcaj_away> maybe there is a better tool we need to find it
[20:48:28] <kbabioch> it is ... either we need to outsource it and/or invest into it (researching other options, fixing it, etc. pp.)
[20:49:15] <mstroeder> I'd also recommend not to run an app which is considered insecure.
[20:49:55] <cboltz> ok, then let me ask an evil question - does someone have a solution that will be ready next week?
[20:50:20] <kbabioch> the evil question is: does it have to be self-hosted / open source
[20:50:26] <mstroeder> Is there an election next week?
[20:50:47] <cboltz> yes, the voting if we want to keep the "openSUSE" name or not
[20:50:48] <tuanpembual> can we install from source?
[20:50:49] <kbabioch> because there is a lot of election stuff ... but i can imagine that not everyone will like to have it hosted somewhere else / by someone else
[20:51:25] <kbabioch> interesting ... that we cannot even provide the platform / tool / infrastructure for such a voting :-/
[20:51:29] <kbabioch> that's actually quite sad
[20:51:30] <tuanpembual> i mean, allow or disallow.
[20:56:51] <cboltz> kbabioch: the "problem" is that very few people need to setup a voting platform, which means only a few people work on it ;-) (development, packaging etc.)
[20:59:20] <cboltz> so - does someone have a solution that will be ready next week? If not, I'll bite the bullet and do the downgrade
[20:59:47] <cboltz> that's clearly nothing we can/will keep long term, but for now it's the best we can do IMHO
[20:59:55] <klein> I have found two options: https://github.com/SmartElect/SmartElect and https://manual.limesurvey.org/Installation_-_LimeSurvey_CE
[21:00:00] <pjessen> agree
[21:00:27] <klein> both aparently are up to date, but, 1 week to setup and learn how to work with it is not enougth
[21:00:50] <mcaj_away> SmartElect looks good let`s test it
[21:02:39] <klein> I think limesurvey is easyer, it is just PHP + MySQL/PostgreSQL
[21:02:41] <kbabioch> hm, so unless someone wants to work on this full-time, i agree with klein ... one week is challenging
[21:03:02] <mcaj_away> well from I point of view there is no way to run 43.2 OS for voting /helios/ ...
[21:03:02] <mcaj_away> What about send an email to board about the situation ..
[21:03:15] <klein> yeah, I think one can setup a server with that running, but, then what? How to create the election, get people to know how to vote and whatever
[21:03:22] <kbabioch> what do you expect the board to do :-)? they can agree wiht us that its bad :-/
[21:03:30] <klein> maybe delay the election?
[21:03:55] <mcaj_away> yes delay until we have the new system up and runnig
[21:04:55] <cboltz> I'm somewhat afraid that delay might be counted in months, so at least for the planned name vote, IMHO that isn't an option
[21:05:17] <cboltz> but we should make clear that the downgrade is only a temporary workaround, and that we need a replacement
[21:05:51] <klein> or, we can agree to ask for 1 month, and in 1 month have a new voting service up and running?
[21:06:07] <mstroeder> openSUSE:infrastructure:elections.opensuse.org/helios-server reports missing python-django-celery but the package seems to be built for Leap 15.1.
[21:07:07] <mcaj_away> im not doing to to downgrade ... what is not good and its violate the SLE between SUSE and openSUSE ...
[21:08:56] <mcaj_away> I tried to fix it .. here:https://build.opensuse.org/project/show/home:mcaj:branches:openSUSE:infrastructure:elections.opensuse.org
[21:08:56] <mcaj_away> it just not work :(
[21:09:11] <mcaj_away> the a problem of old code
[21:09:29] <kbabioch> btw: we have a similar situation with piwik / matomo ... its also an outdated versoin (included into many opensuse web assets) ... would also need some love :-/
[21:09:44] <cboltz> agreed in general, but python didn't change that much (at least as long as you stay with py2)
[21:10:10] <cboltz> so I'd hope it should be fixable without too much effort
[21:10:19] <cboltz> but probably not within a week...
[21:11:40] <mstroeder> To me it looks like outdated dependencies.
[21:12:11] <klein> can we run a docker with older python version ?
[21:12:37] * klein have some extremeGoHorse feelings :-)
[21:13:32] <mcaj_away> docker .. maybe that can be way ...
[21:13:50] <mstroeder> It's not the Python version. Leap 15.1 has Python 2.7.x and this should not be an issue.
[21:14:18] <kbabioch> let's run some random docker container :-/ -> https://hub.docker.com/r/acspri/limesurvey/
[21:14:41] <klein> so it could be a way to run it from source or inside a virtual_env
[21:15:00] <klein> I don't like to just run random containers :-(
[21:15:11] <mstroeder> Yeah, let's ignore basic container security / hygiene...not!
[21:15:31] <cboltz> same for me - I'd even prefer cat /dev/random | sudo bash ;-)
[21:15:49] <klein> LOL
[21:18:02] <cboltz> so - do we have any short-term solutions for next week?
[21:18:23] <cboltz> as much as I (and everybody) hate the idea, I'd volunteer to do the downgrade
[21:18:51] <cboltz> obviously we should make it very clear that we need a replacement ASAP, but I'd hate to delay the voting
[21:18:57] <mcaj_away> I would say let`s try a docketr 42.3 image in 15.1...
[21:19:19] <klein> that may be the easyest path
[21:19:33] <cboltz> how is that better / more secure than a 42.3 VM?
[21:19:58] <cboltz> the part that is exposed to the outside will still be the same
[21:19:59] <klein> well... its the same, but you do not need to do the downgrade process :-)
[21:20:19] <cboltz> the downgrade is quite easy - actually I already tried it in my local test VM
[21:20:26] <cboltz> so no need to worry about that
[21:20:37] <klein> well... ok for me then
[21:21:06] <cboltz> ok, then I'll do it later today or tomorrow
[21:21:11] <klein> just one quetion, how much time this meeting use to have?
[21:21:13] <klein> *question
[21:21:31] <cboltz> depends on how many topics we discuss ;-)
[21:21:40] <mcaj_away> personally I will NOT do the downgrade until I will got dirrect order from my team lead ... as we have this so call SLA
[21:21:46] <cboltz> I'd say an hour on average
[21:22:26] <mcaj_away> I will need to go in a few minutes ... (8)
[21:22:44] <cboltz> mcaj_away: understood, and I hate it as much as you hate it - but it's the least bad (avoiding "best") option I see
[21:23:35] <mcaj_away> Just a question what happen whetre there is not tool next week to vote ?
[21:24:22] <cboltz> well, two things:
[21:24:29] <cboltz> - there won't be a voting (obviously)
[21:24:37] <mcaj_away> will we mayve somebody in the board reliase that thare is a problem to run out of date tool
[21:25:08] <cboltz> - we'll get lots of funny comments about not being able to run an election once more - that's a tradition I'd prefer not to continue ;-)
[21:25:53] <pjessen> yeah
[21:26:15] <cboltz> I'll bring up this topic in the next board meeting (in 35 minutes)
[21:26:43] <mcaj_away> do please is a big problem from my point of view ...
[21:26:46] <cboltz> but I'm somewhat sure that - at least for the voting next week - downgrading 42.3 will be prefered over delaying the voting
[21:27:17] <tuanpembual> and this week, EC team will meet at openSUSE Asia Summit.
[21:27:18] <mcaj_away> election should be done on the secure SW and then is trusted by everybody ...
[21:27:20] <cboltz> I'll also make it clear that it is only a temporary solution
[21:28:30] <cboltz> tuanpembual: some board members will also be there, feel free to grab them when discussing this topic
[21:29:00] <tuanpembual> yeps, it will part of community meetup agenda.
[21:29:51] <mcaj_away> its easy to set up a tool but keep it up to date for years is the hard part ...
[21:30:12] <tuanpembual> I need to go.
[21:30:45] <tuanpembual> will read this log latter. thanks everyone and Good morning
[21:31:28] <cboltz> mcaj_away: agreed, but in this case not even the setup was that easy IIRC ;-)
[21:32:06] <cboltz> anyway - maybe we should switch to a completely different topic you might like more?
[21:32:21] <cboltz> we didn't have a face2face meeting for more than a year
[21:32:45] <cboltz> therefore I'd like to propose to have one (probably in Nuremberg)
[21:33:22] <cboltz> maybe in November?
[21:33:29] <mcaj_away> kbabioch: ^^ ?
[21:33:49] <cboltz> I already warned kbabioch about that idea before ;-)
[21:34:04] <pjessen> didn't we talk about it last week?
[21:34:06] <kbabioch> well, i've talked to roland ... we have a budget
[21:34:28] <kbabioch> and we are also allowed to use the suse facilities (conference rooms)
[21:34:34] <kbabioch> and go out for some pizza, etc.
[21:34:43] <kbabioch> but we would need an estimation, i.e. how many people are going to join?
[21:36:05] <cboltz> the last meeting was maybe 20 people (not all of them full-time), and at least 10 of them from Nuremberg
[21:36:13] <kbabioch> so how many people do we expect to join this time?
[21:36:18] <kbabioch> will it be this many again?
[21:36:20] <pjessen> if you are asking for a show of hands, count me in
[21:36:39] <cboltz> I'll also be there
[21:36:55] <mcaj_away> what about an email to heroes@o.o and ask them/us
[21:36:59] <kbabioch> should i maybe write to the mailing list and try to get a better understanding on who will be coming?
[21:37:05] <mcaj_away> I will definitly go
[21:37:10] <pjessen> yeah, that's a good idea
[21:37:15] <cboltz> yes, please ask on the mailinglist
[21:37:24] <kbabioch> okay
[21:37:43] <cboltz> should we propose a date to avoid endless discussions?
[21:38:03] <kbabioch> makes sense
[21:38:05] <mcaj_away> around middle of November ?
[21:38:11] <klein> I'm in
[21:38:25] <cboltz> sounds good
[21:38:46] <pjessen> 16/17 nov ?
[21:39:05] <cboltz> yes, + dinner on 15th ;-)
[21:39:10] <mcaj_away> looks good
[21:40:12] <kbabioch> so, how was this usually done?
[21:40:14] <kbabioch> two full days?
[21:40:19] <kbabioch> how many nights did you stay?
[21:40:35] <pjessen> two nights, fri-sat and sat-sun
[21:41:29] <kbabioch> okay, and dinner on friday then?
[21:41:47] <kbabioch> and "hackathon" on saturday & sunday?
[21:41:49] <mcaj_away> Yes that need it ;_
[21:42:57] <cboltz> the funny thing is that we used more whiteboards than keyboards last time - but that's not set into stone
[21:42:59] <mcaj_away> Heroes hackathon 2019
[21:43:57] <mcaj_away> let`s see ...
[21:44:45] <kbabioch> okay, will send mail
[21:44:57] <mcaj_away> I need to go now it was a long day ...
[21:45:06] <mcaj_away> CU tomorrow ...
[21:48:05] <cboltz> does someone have another topic, or should we close the meeting?
[21:49:12] <cboltz> looks like we can close the meeting ;-)
[21:49:19] <cboltz> thanks everybody for joining!
[21:49:51] <kbabioch> cu
[21:50:14] <klein> cu
    (1-1/1)