Project

General

Profile

2019-10-01-heroes-meeting.txt

IRC meeting log - cboltz, 2019-10-01 22:19

 
1
2019-10-01 #opensuse-admin heroes meeting
2
[20:00:57] <cboltz> Hi everybody!
3
[20:01:03] <cboltz> who is here for the heroes meeting?
4
[20:01:57] <mstroeder> michael@ae-dir.com
5
[20:02:44] <klein> rklein@suse.com
6
[20:02:59] <cboltz> hi!
7
[20:03:11] <cboltz> (no need to show up by mail address ;-)
8
[20:03:17] <tuanpembual> hi
9
[20:03:39] <tuanpembual> me for heroes meeting
10
[20:03:51] * klein newbie, don't know the "protocol"
11
[20:04:33] <cboltz> well, it's a meeting ;-)
12
[20:04:45] <cboltz> officially we have a schedule, see https://progress.opensuse.org/issues/56435
13
[20:05:13] <cboltz> but in practise we don't handle it that strict and just discuss what needs to be discussed ;-)
14
[20:05:24] <cboltz> the only somewhat fixed item is:
15
[20:05:32] <cboltz> does someone from the community have a question?
16
[20:06:16] <cboltz> (that often results in silence, but I'm happy about everybody who surprises me/us, so please ask ;-)
17
[20:06:41] <pjessen> I'm back
18
[20:06:46] <tuanpembual> hi pjessen
19
[20:06:49] <tuanpembual> :D
20
[20:06:55] <cboltz> hi!
21
[20:07:35] <cboltz> looks like we can continue with the next topic - status reports about everything
22
[20:07:40] <klein> do we have a current diagram of our infra and a list of "mission critical systems" that need to be running?
23
[20:07:59] * kbabioch is also here
24
[20:08:20] <cboltz> not exactly
25
[20:08:37] <cboltz> we have the monitoring, and we also have the list of machines in salt (pillar/id/*)
26
[20:09:25] <cboltz> and maybe a few people have a list of mission critical systems in /dev/brain ;-) - but nothing written down AFAIK
27
[20:09:39] <klein> ok, I will see the pillars and then ask more questions in the next meeting.
28
[20:10:21] <cboltz> you can also ask earlier (here or on the mailinglist), no need to wait a month ;-)
29
[20:10:37] <klein> ok
30
[20:11:15] <mstroeder> Missing documentation make it really hard for volunteers to fix something on existing systems. Diagrams can be easily coded with http://blockdiag.com for systems and networks.
31
[20:11:52] <cboltz> I fully agree that missing documentation makes it hard
32
[20:12:13] <cboltz> but I slightly doubt that we need diagrams - our setup isn't that complicated IMHO ;-)
33
[20:12:17] <klein> but too much docs makes it harder too :-)
34
[20:12:54] <cboltz> (typically $service = haproxy + actual (web) server + maybe database server)
35
[20:13:10] <mstroeder> There is much room between no docs and too many docs.
36
[20:13:24] <cboltz> I've rarely seen "too many docs" ;-)
37
[20:14:36] <cboltz> since we are already discussing this:
38
[20:14:52] <cboltz> we have https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Machines (most likely outdated)
39
[20:14:58] <cboltz> and we have some data in salt pillar/id/*
40
[20:15:13] <mstroeder> Without blaming somebody I'd like to point out the Let's Encrypt fixes raised on the mailing list during the last days. The custom stuff and its dependencies are too hard to find out while in a hurry.
41
[20:15:52] <cboltz> no objections ;-)
42
[20:16:17] <kbabioch> mstroeder: yup, its heavily customized (hook script for dehydrated) ... but to be fair: not sure how else to do it, since we're using freeipa for dns management and want to have wildcard certs (i.e. we need dns authentication)
43
[20:17:10] <kbabioch> and it took 2 persons 1,5 hours or so, so afterall its not too bad, since we also had to "hack" our way into the machine (no credentials were documented ;-))
44
[20:17:12] <klein> I dont see dehydated as a bad thing... it has a post/pre hook feature, we just need to document it better
45
[20:18:31] <kbabioch> let's put it this way (and this is a serious quesiton): has anyone ever worked in an environment where documentation was good/sufficient? because everyone always wants to have it, but in my experience its always a challenge
46
[20:19:09] <kbabioch> in some places there wasn't any, in some other places there is outdated documentation (which is even worse) ... but never seen a place, where everything was perfect in terms of documentation ...
47
[20:19:15] <klein> sorry, I mean just have a README in /root telling that you should use dehydrated and kinit, nothing that fancy
48
[20:19:40] <klein> the rest, one can figure by himself
49
[20:20:28] <klein> more than that, will make us ending up with outdated docs
50
[20:21:26] <cboltz> personally I'd prefer to document that with some pkg.installed in salt ;-) (maybe with some comments added) but in general I agree that some basic docs would be useful
51
[20:21:43] <kbabioch> i agree ... but we're currently even struggling with figuring out if machines are "reboot_safe" ... i have my doubts if we will have time/bandwith to work on those kin of docs
52
[20:22:08] <pjessen> me too
53
[20:22:27] <cboltz> well, do it like you do it with reboot_safe - whenever you find out something worth documenting, document it
54
[20:22:30] <klein> maybe that list of servers and what they do (a short description) would help to know wich ones are safe and the ones that need more love
55
[20:22:47] <cboltz> even if it's just a comment in a salt file, that's much better than nothing
56
[20:23:17] <mcaj_away> Hi all
57
[20:23:20] <klein> yeah, maybe we can find a way to have good pillars or someting that moves with the living infra
58
[20:23:37] <klein> and not some wiki that no one updates
59
[20:23:58] <cboltz> can you read my /dev/brain? I just wanted to propose to move the content of the machine pages into pillar/id/ ;-)
60
[20:24:11] * mcaj_away is on a pub wifi ...
61
[20:24:29] <pjessen> hi martin
62
[20:24:30] <cboltz> that's more likely to get updated than wiki pages, and has a smaller risk of getting out of sync
63
[20:24:37] <klein> cboltz: :-) that makes perfect sense
64
[20:25:11] <mcaj_away> pjessen: hi, a pub wifi had some packetloss, but now is fine
65
[20:25:41] <cboltz> give me some days to come up with a proof-of-concept merge request showing it for a few hosts
66
[20:26:13] <cboltz> I'll send the link to the mailinglist, and if nobody complains, do it for more machines
67
[20:26:16] <tuanpembual> I dont have access to pillar or gitlab.
68
[20:26:18] <tuanpembual> :D
69
[20:26:43] <cboltz> that's something we'll need to fix ;-)
70
[20:26:47] <tuanpembual> who can I request to this?
71
[20:27:03] <mcaj_away> we can fix that just send an email to admin@opensuse.org
72
[20:27:10] <mcaj_away> I can fix it ...
73
[20:27:43] <tuanpembual> oke mcaj_away, will send email. thanks
74
[20:28:04] <tuanpembual> I do usually write tec doc everyday.
75
[20:28:18] <tuanpembual> using markdown at private gitlab.
76
[20:28:27] <tuanpembual> maybe I can help more.
77
[20:28:56] <tuanpembual> I want report my progress.
78
[20:29:41] <cboltz> go ahead ;-)
79
[20:29:43] <tuanpembual> https://progress.opensuse.org/issues/27720 last status, need pointing domain.
80
[20:30:22] <tuanpembual> that all.
81
[20:31:23] <cboltz> so - who can setup progress-test.o.o as CNAME login2.o.o for tuanpembual (ideally _now_ ;-) ?
82
[20:32:20] * cboltz doesn't have permissions to add DNS entries
83
[20:32:27] <mcaj_away> tuanpembual: try gitlab your account is not "unblock"
84
[20:32:40] <mcaj_away> s/not/now
85
[20:32:55] <tuanpembual> sure.
86
[20:33:34] <kbabioch> i can do it now
87
[20:34:46] <tuanpembual> thank mcaj_away, it work.
88
[20:35:09] <mcaj_away> see this is the admin super power ;)
89
[20:35:39] <cboltz> mcaj_away: the second half is/was to give tuanpembual developer access to https://gitlab.infra.opensuse.org/infra/salt/ - which I did before tuanpembual tested ;-)
90
[20:36:06] <kbabioch> progress-test.opensuse.org is setup ... might take a while for the zone transfer to happen
91
[20:36:13] <cboltz> (AFAIK that repo is only accessible for project members, not for random gitlab users ;-)
92
[20:36:18] <cboltz> thanks!
93
[20:36:21] <kbabioch> but actually it already works
94
[20:37:09] <cboltz> it still needs some config in haproxy
95
[20:37:10] <tuanpembual> I will continue tuning new redmine.
96
[20:37:24] <tuanpembual> *pointing domain etc.
97
[20:37:31] <mcaj_away> done
98
[20:37:38] <tuanpembual> thank kbabioch
99
[20:38:25] <mcaj_away> btw did you already speak about helios ? https://progress.opensuse.org/issues/57104
100
[20:38:39] <cboltz> not yet, but we should ;-)
101
[20:38:40] <kbabioch> nope, we didn't
102
[20:39:01] <cboltz> my proposal is:
103
[20:39:21] <cboltz> - as a quickfix (because we need it next week) downgrade to 42.3
104
[20:39:40] <cboltz> - longer term, either upgrade everything or find a replacement
105
[20:40:01] <kbabioch> downgrading will also mean to run an outdated / unsupported machine ... not exactly a good idea :-/
106
[20:40:33] <mcaj_away> yes we have this SLA we can not have out of support OS
107
[20:40:38] <cboltz> I know (and hate it as much as everybody else), but not having an election tool when we need it is not better :-/
108
[20:41:01] <mstroeder> Error message "No module named django" means that Django is not installed.
109
[20:41:07] <mcaj_away> we should maybe send a request do comunity, developers or board
110
[20:41:20] <mcaj_away> about fix the tool or find new one
111
[20:41:59] <mstroeder> Or Django module package is not readable due to missing AppArmor rules...
112
[20:42:02] <mcaj_away> the problem with it is that is out of date and not compatible and secure with currect python verison
113
[20:42:32] <cboltz> mstroeder: the funny thing is that Django _is_ installed (at least as far as I can see), and IIRC there isn't an AppArmor profile on it
114
[20:43:31] <cboltz> which means the error message is IMHO somewhat misleading
115
[20:43:42] <cboltz> mcaj_away: did you look into more details?
116
[20:43:58] <mstroeder> Is it a Py2/Py3 compat issue?
117
[20:44:37] <mcaj_away> I spoke wiht Tomas Chvatal and he told me two years ago the tool was problematic, now  days it just not secure and fixible without changes on srource code/ upstream
118
[20:45:34] <mcaj_away> you can look here https://heliosvoting.org/ and here https://github.com/benadida/helios-server
119
[20:47:03] * kbabioch thinks that we shouldn't run an election tool if we cannot manage to package, maintain and operate it :-/
120
[20:47:24] <mcaj_away> and there is no update on upstream
121
[20:47:27] <cboltz> yeah, but not running an election tool also isn't an option...
122
[20:48:08] <mcaj_away> maybe there is a better tool we need to find it
123
[20:48:28] <kbabioch> it is ... either we need to outsource it and/or invest into it (researching other options, fixing it, etc. pp.)
124
[20:49:15] <mstroeder> I'd also recommend not to run an app which is considered insecure.
125
[20:49:55] <cboltz> ok, then let me ask an evil question - does someone have a solution that will be ready next week?
126
[20:50:20] <kbabioch> the evil question is: does it have to be self-hosted / open source
127
[20:50:26] <mstroeder> Is there an election next week?
128
[20:50:47] <cboltz> yes, the voting if we want to keep the "openSUSE" name or not
129
[20:50:48] <tuanpembual> can we install from source?
130
[20:50:49] <kbabioch> because there is a lot of election stuff ... but i can imagine that not everyone will like to have it hosted somewhere else / by someone else
131
[20:51:25] <kbabioch> interesting ... that we cannot even provide the platform / tool / infrastructure for such a voting :-/
132
[20:51:29] <kbabioch> that's actually quite sad
133
[20:51:30] <tuanpembual> i mean, allow or disallow.
134
[20:56:51] <cboltz> kbabioch: the "problem" is that very few people need to setup a voting platform, which means only a few people work on it ;-)  (development, packaging etc.)
135
[20:59:20] <cboltz> so - does someone have a solution that will be ready next week? If not, I'll bite the bullet and do the downgrade
136
[20:59:47] <cboltz> that's clearly nothing we can/will keep long term, but for now it's the best we can do IMHO
137
[20:59:55] <klein> I have found two options: https://github.com/SmartElect/SmartElect and https://manual.limesurvey.org/Installation_-_LimeSurvey_CE
138
[21:00:00] <pjessen> agree
139
[21:00:27] <klein> both aparently are up to date, but, 1 week to setup and learn how to work with it is not enougth
140
[21:00:50] <mcaj_away> SmartElect looks good let`s test it
141
[21:02:39] <klein> I think limesurvey is easyer, it is just PHP + MySQL/PostgreSQL
142
[21:02:41] <kbabioch> hm, so unless someone wants to work on this full-time, i agree with klein ... one week is challenging
143
[21:03:02] <mcaj_away> well from I point of view there is no way to run 43.2 OS for voting /helios/ ...
144
[21:03:02] <mcaj_away> What about send an email to board about the situation ..
145
[21:03:15] <klein> yeah, I think one can setup a server with that running, but, then what? How to create the election, get people to know how to vote and whatever
146
[21:03:22] <kbabioch> what do you expect the board to do :-)? they can agree wiht us that its bad :-/
147
[21:03:30] <klein> maybe delay the election?
148
[21:03:55] <mcaj_away> yes delay until we have the new system up and runnig
149
[21:04:55] <cboltz> I'm somewhat afraid that delay might be counted in months, so at least for the planned name vote, IMHO that isn't an option
150
[21:05:17] <cboltz> but we should make clear that the downgrade is only a temporary workaround, and that we need a replacement
151
[21:05:51] <klein> or, we can agree to ask for 1 month, and in 1 month have a new voting service up and running?
152
[21:06:07] <mstroeder> openSUSE:infrastructure:elections.opensuse.org/helios-server reports missing python-django-celery but the package seems to be built for Leap 15.1.
153
[21:07:07] <mcaj_away> im not doing to to downgrade ... what is not good and its violate the SLE between SUSE and openSUSE ...
154
[21:08:56] <mcaj_away> I tried to fix it .. here:https://build.opensuse.org/project/show/home:mcaj:branches:openSUSE:infrastructure:elections.opensuse.org
155
[21:08:56] <mcaj_away> it just not work :(
156
[21:09:11] <mcaj_away> the a problem of old code
157
[21:09:29] <kbabioch> btw: we have a similar situation with piwik / matomo ... its also an outdated versoin (included into many opensuse web assets) ... would also need some love :-/
158
[21:09:44] <cboltz> agreed in general, but python didn't change that much (at least as long as you stay with py2)
159
[21:10:10] <cboltz> so I'd hope it should be fixable without too much effort
160
[21:10:19] <cboltz> but probably not within a week...
161
[21:11:40] <mstroeder> To me it looks like outdated dependencies.
162
[21:12:11] <klein> can we run a docker with older python version ?
163
[21:12:37] * klein have some extremeGoHorse feelings :-)
164
[21:13:32] <mcaj_away> docker .. maybe that can be way  ...
165
[21:13:50] <mstroeder> It's not the Python version. Leap 15.1 has Python 2.7.x and this should not be an issue.
166
[21:14:18] <kbabioch> let's run some random docker container :-/ -> https://hub.docker.com/r/acspri/limesurvey/
167
[21:14:41] <klein> so it could be a way to run it from source or inside a virtual_env
168
[21:15:00] <klein> I don't like to just run random containers :-(
169
[21:15:11] <mstroeder> Yeah, let's ignore basic container security / hygiene...not!
170
[21:15:31] <cboltz> same for me - I'd even prefer   cat /dev/random | sudo bash   ;-)
171
[21:15:49] <klein> LOL
172
[21:18:02] <cboltz> so - do we have any short-term solutions for next week?
173
[21:18:23] <cboltz> as much as I (and everybody) hate the idea, I'd volunteer to do the downgrade
174
[21:18:51] <cboltz> obviously we should make it very clear that we need a replacement ASAP, but I'd hate to delay the voting
175
[21:18:57] <mcaj_away> I would say let`s try a docketr 42.3 image in 15.1...
176
[21:19:19] <klein> that may be the easyest path
177
[21:19:33] <cboltz> how is that better / more secure than a 42.3 VM?
178
[21:19:58] <cboltz> the part that is exposed to the outside will still be the same
179
[21:19:59] <klein> well... its the same, but you do not need to do the downgrade process :-)
180
[21:20:19] <cboltz> the downgrade is quite easy - actually I already tried it in my local test VM
181
[21:20:26] <cboltz> so no need to worry about that
182
[21:20:37] <klein> well... ok for me then
183
[21:21:06] <cboltz> ok, then I'll do it later today or tomorrow
184
[21:21:11] <klein> just one quetion, how much time this meeting use to have?
185
[21:21:13] <klein> *question
186
[21:21:31] <cboltz> depends on how many topics we discuss ;-)
187
[21:21:40] <mcaj_away> personally I will NOT do the downgrade until I will got dirrect order from my team lead ... as we have this so call SLA
188
[21:21:46] <cboltz> I'd say an hour on average
189
[21:22:26] <mcaj_away> I will need to go in a few minutes ... (8)
190
[21:22:44] <cboltz> mcaj_away: understood, and I hate it as much as you hate it - but it's the least bad (avoiding "best") option I see
191
[21:23:35] <mcaj_away> Just a question what happen whetre there is not tool next week to vote ?
192
[21:24:22] <cboltz> well, two things:
193
[21:24:29] <cboltz> - there won't be a voting (obviously)
194
[21:24:37] <mcaj_away> will we mayve somebody in the board reliase that thare is a problem to run out of date tool
195
[21:25:08] <cboltz> - we'll get lots of funny comments about not being able to run an election once more - that's a tradition I'd prefer not to continue ;-)
196
[21:25:53] <pjessen> yeah
197
[21:26:15] <cboltz> I'll bring up this topic in the next board meeting (in 35 minutes)
198
[21:26:43] <mcaj_away> do please is a big problem from my point of view ...
199
[21:26:46] <cboltz> but I'm somewhat sure that - at least for the voting next week - downgrading 42.3 will be prefered over delaying the voting
200
[21:27:17] <tuanpembual> and this week, EC team will meet at openSUSE Asia Summit.
201
[21:27:18] <mcaj_away> election should be done on the secure SW and then is trusted by everybody ...
202
[21:27:20] <cboltz> I'll also make it clear that it is only a temporary solution
203
[21:28:30] <cboltz> tuanpembual: some board members will also be there, feel free to grab them when discussing this topic
204
[21:29:00] <tuanpembual> yeps, it will part of community meetup agenda.
205
[21:29:51] <mcaj_away> its easy to set up a tool but keep it up to date for years is the hard part ...
206
[21:30:12] <tuanpembual> I need to go.
207
[21:30:45] <tuanpembual> will read this log latter. thanks everyone and Good morning
208
[21:31:28] <cboltz> mcaj_away: agreed, but in this case not even the setup was that easy IIRC ;-)
209
[21:32:06] <cboltz> anyway - maybe we should switch to a completely different topic you might like more?
210
[21:32:21] <cboltz> we didn't have a face2face meeting for more than a year
211
[21:32:45] <cboltz> therefore I'd like to propose to have one (probably in Nuremberg)
212
[21:33:22] <cboltz> maybe in November?
213
[21:33:29] <mcaj_away> kbabioch: ^^ ?
214
[21:33:49] <cboltz> I already warned kbabioch about that idea before ;-)
215
[21:34:04] <pjessen> didn't we talk about it last week?
216
[21:34:06] <kbabioch> well, i've talked to roland ... we have a budget
217
[21:34:28] <kbabioch> and we are also allowed to use the suse facilities (conference rooms)
218
[21:34:34] <kbabioch> and go out for some pizza, etc.
219
[21:34:43] <kbabioch> but we would need an estimation, i.e. how many people are going to join?
220
[21:36:05] <cboltz> the last meeting was maybe 20 people (not all of them full-time), and at least 10 of them from Nuremberg
221
[21:36:13] <kbabioch> so how many people do we expect to join this time?
222
[21:36:18] <kbabioch> will it be this many again?
223
[21:36:20] <pjessen> if you are asking for a show of hands, count me in
224
[21:36:39] <cboltz> I'll also be there
225
[21:36:55] <mcaj_away> what about an email to heroes@o.o and ask them/us
226
[21:36:59] <kbabioch> should i maybe write to the mailing list and try to get a better understanding on who will be coming?
227
[21:37:05] <mcaj_away> I will definitly go
228
[21:37:10] <pjessen> yeah, that's a good idea
229
[21:37:15] <cboltz> yes, please ask on the mailinglist
230
[21:37:24] <kbabioch> okay
231
[21:37:43] <cboltz> should we propose a date to avoid endless discussions?
232
[21:38:03] <kbabioch> makes sense
233
[21:38:05] <mcaj_away> around middle of November  ?
234
[21:38:11] <klein> I'm in
235
[21:38:25] <cboltz> sounds good
236
[21:38:46] <pjessen> 16/17 nov ?
237
[21:39:05] <cboltz> yes, + dinner on 15th ;-)
238
[21:39:10] <mcaj_away> looks good
239
[21:40:12] <kbabioch> so, how was this usually done?
240
[21:40:14] <kbabioch> two full days?
241
[21:40:19] <kbabioch> how many nights did you stay?
242
[21:40:35] <pjessen> two nights, fri-sat and sat-sun
243
[21:41:29] <kbabioch> okay, and dinner on friday then?
244
[21:41:47] <kbabioch> and "hackathon" on saturday & sunday?
245
[21:41:49] <mcaj_away> Yes that need it ;_
246
[21:42:57] <cboltz> the funny thing is that we used more whiteboards than keyboards last time - but that's not set into stone
247
[21:42:59] <mcaj_away> Heroes hackathon 2019
248
[21:43:57] <mcaj_away> let`s see ...
249
[21:44:45] <kbabioch> okay, will send mail
250
[21:44:57] <mcaj_away> I need to go now it was a long day ...
251
[21:45:06] <mcaj_away> CU tomorrow ...
252
[21:48:05] <cboltz> does someone have another topic, or should we close the meeting?
253
[21:49:12] <cboltz> looks like we can close the meeting ;-)
254
[21:49:19] <cboltz> thanks everybody for joining!
255
[21:49:51] <kbabioch> cu
256
[21:50:14] <klein> cu