2020-09-01 #opensuse-admin - heroes meeting [20:00:30] Hi everybody, and welcome to the heroes meeting! [20:00:39] the topics are on https://progress.opensuse.org/issues/69589 [20:01:17] <[Lemmy]> 'lo [20:01:56] hello [20:02:21] hello [20:03:22] does someone from the community have questions? [20:03:48] yo [20:05:28] looks like nobody has questions, so let's start with the status report round [20:05:37] I have two things [20:05:44] 1 - provo-mirror [20:05:55] it's still outdated, rsync keeps running and running [20:06:33] additionally I found out that specific files let rsync crash (bugreport opened) - for now, the workaround is not to use --compress [20:06:42] <[Lemmy]> ...excuse the question, what is provo-mirror? [20:06:49] provo-mirror.opensuse.org [20:06:53] <[Lemmy]> ah ok [20:07:17] 2 - release notes on doc.opensuse.org [20:07:19] maybe run more processes with smaller chunks of data? [20:08:38] might be worth a try, but the general issue is network speed (I see IIRC ~6 M/s disk write in iotop) [20:09:01] do you have an idea if provo-mirror only has a 100 MBit connection and/or a bandwidth limit per IP? [20:10:54] (at least that's the only valid explanation I can imagine for the quite slow sync) [20:12:18] <[Lemmy]> have you checked the network with iperf? [20:12:33] <[Lemmy]> that, and bonnie++ on the storage [20:13:08] <[Lemmy]> also, is there anything suspicious about the cpu load when a sync is running? [20:13:33] I didn't do much debugging yet, but I'm quite sure that the storage is not the issue ;-) and the CPUs are bored [20:13:46] <[Lemmy]> then run an iperf [20:14:22] sounds like I have something on my TODO list for later ;-) [20:15:34] so, next report from me: [20:15:39] 2 - release notes on doc.opensuse.org [20:16:10] with the rpm compression switched to zstd, the script on community.i.o.o wasn't able to extract the tumbleweed release notes anymore [20:16:24] and to make things more funny, this broke openQA [20:17:20] luckily lcp had the release notes mostly ready on pinot.i.o.o, so I "only" had to do a few hotfixes to get them actually deployed there (see the MR in gitlab) [20:17:44] I adjusted haproxy so that (only) the tumbleweed release notes are now served from pinot [20:17:59] there are some things left on the TODO list before we can fully switch over [20:18:32] the most funny thing was that I had the fix in place before dimstar finished writing the ticket :-) [20:18:32] ha.. ha... [20:19:34] lcp: even if some things are left - thanks a lot for the work you've done on the release notes hosting! [20:19:42] yeah, the doc stuff is currently generated with a perl script, and will be moved over to jekyll when the pr is merged [20:20:25] that's it from me [20:20:40] who else has a status report? [20:20:42] mailman3 server is set up and ready, with the exception of mails directed at `lists.opensuse.org` not reaching the machine, which is a problem for a mailing list server [20:21:24] I also have to prep the instructions for uyuni guys to switch over the domain [20:21:40] then we can migrate :P [20:21:57] <[Lemmy]> since the mail topic is up... [20:22:00] question, did we gad mailman2 in place before? [20:22:13] *did we had [20:22:25] no, the current setup uses mlmmj [20:22:26] no, we are migrating from mlmmj to mailman3 [20:22:48] mlmmj + mhonarc to be precise [20:22:49] hh, ok [20:22:54] hello, will be joining in about 15min [20:23:06] did we migrated from mhonarc the files? [20:23:26] i need to check that to be able to migrate the internal mailman on suse 😬 [20:23:36] but, will ask you later [20:23:41] that's gonna be migrated as everything else is migrated [20:23:46] to not kill the meeting [20:23:59] currently we have delivery issues, which has to be fixed before we can think about moving over ;) [20:24:42] we do have a precise plan for migration though https://progress.opensuse.org/issues/70018 [20:25:31] (it would have been easier with mm2, since there are scripts in mm3 to do this easily) [20:25:39] matrix had its login stuff set up, but there is some miscomunication between it and ipsilon, so it doesn't get properties it needs to register the user [20:26:22] I assigned bmwiedemann to that issue, but I don't know what part is to blame for why it doesn't fully work [20:27:11] bmwidemann is on vacation [20:27:25] noted [20:27:45] can't blame him, it's a good time to take a break ;) [20:27:52] I did some migration tests for vb -> discourse transition, it takes 3 days to complete. I also tested if it's possible to fix the encoding errors, and there is no easy way, and I'm thinking what the right course of action for that would be in that case [20:28:29] maybe a bunch of sed on the dump file? [20:29:09] that sounds more scary than keeping the broken encoding ;-) [20:29:21] the issue is that there is a lot of stuff that's encoded right, and I have seen how badly that can be garbled with sed [20:30:27] maybe we can just remove posts older than may 2011, they are no longer relevant to anything that's supported more than likely (although that's a very drastic way to deal with an encoding issue) [20:31:03] in that case, I'd prefer to keep the broken encoding ;-) [20:31:32] alright, that makes sense [20:32:03] I believe that's it from me [20:33:37] thanks! [20:33:42] who else has a status report? [20:35:24] no update from me.will close some ticket abiut progress bug this week. [20:36:29] ok, thanks [20:37:00] I guess pjessen isn't here yet, so let's have a look at some old tickets until he arrives [20:39:16] (whoever wonders about the status of an old ticket, please shout out the number ;-) [20:40:17] I assume that since https://progress.opensuse.org/issues/70513 was fixed, it could be closed? [20:40:47] right - looks like I only added a comment and forgot to change the status [20:40:49] I'll close it [20:41:06] below that there are 3 issues about repos not working [20:41:34] https://progress.opensuse.org/issues/70465 https://progress.opensuse.org/issues/70447 https://progress.opensuse.org/issues/70432 [20:41:55] (they are also all private for some reason) [20:43:21] 70447 looks like a broken mirror [20:43:40] am back. [20:44:35] just in time - lcp picked up some (somewhat) old tickets about non-working repos ;-) [20:44:57] (see the links some above) [20:45:01] hello everybody. sorry for being so late, could not join earlier [20:45:02] alright, I guess I didn't read into them hard enough, so only 2 of them were about the downtime at that time? [20:46:06] do we have any idea why they were unreachable? [20:46:09] also hello [20:47:01] lcp: mirrors that were unreachable? [20:47:48] well, I assume if it was also unreachable from the browser, it wasn't just the repos [20:48:35] there isn't much we can do - a mirror can off-line, come back later. [20:49:08] if it is a more permanent state, mirrorbrain will pick it up and stop handing out that mirror to requests [20:49:44] alright, gotcha [20:50:22] 70447 - sounds like a user problem, unless we had some issue with dowenload.o.o at that time [20:50:37] so I guess we could ask for feedback on the two that haven't responded if the repos came back (even if a week late), and depending on the answer close them [20:51:06] 70465 - crystal ball needed. [20:51:44] lcp: yes, when it has been a week, asking if the problem persists is a good start. [20:52:12] pjessen: the interesting thing in 70447 is that it seems to be time-dependent (the reporter says it works in the evening/night, and breaks in the afternoon) [20:52:46] and HTTP/0.9 also sounds a bit unusual [20:53:04] cboltz: yeah. but we ought to see that one in the log [20:53:41] if it came from us. [20:54:00] I'll ask to check zypper.log for the mirror used [20:56:58] the real issue in e.g. 70447 is that we have little means to investigate. no other users have reported the same, which is why I think "user problem". [20:58:37] * jdsn has to leave now and is for audio meetings (video ois k if audio meetings have downsides) [20:58:50] do we want to move on? [20:59:45] makes sense - we can do more ticket wrangling at the end of the meeting if we have some time and motivation left ;-) [20:59:52] so - next topic: [21:00:05] mx1/2: anti-spam / anti-virus fine tuning [21:00:16] pjessen: I guess that's your topic? [21:00:30] cboltz: yeah - which prompts me to quickly mention our mail-server migration. [21:00:59] basically a major success, nobody noticed. :-) [21:01:36] yeah, "no complaints" is the best result you can get for a migration ;-) [21:02:44] as for fine-tuning the spam/virus filter - personally I intend to do some minor adjustment that will benefit everyone, but otherwise [21:03:06] it is a full-time job to maintain a filter for suit a group of some 400-500 people. [21:04:49] <[Lemmy]> have you guys ever looked into greylisting? [21:05:06] [Lemmy]: it is active. [21:05:25] <[Lemmy]> hm [21:06:08] we only apply it for poorly configured hosts [21:06:21] <[Lemmy]> all i know is that about half of the spam that actually makes it past my own spamassassin was sent through mx*.o.o to my opensuse address [21:07:23] <[Lemmy]> also: the abuse whois for suse.* points at an email address that does a **lot** of bounces [21:07:51] <[Lemmy]> to the effect that services like spamcop don't even bother sending reports anymore [21:08:11] [Lemmy]: the latter is perhaps a topic SUSE-IT, dunno. [21:08:41] <[Lemmy]> there's actually a ticket, 55838 [21:08:42] [Lemmy]: really: the abuse whois for suse.* ?? [21:09:08] ok, good that its tracked already [21:09:40] <[Lemmy]> well, spamcop analyses the mail headers of a spam mail, and arrives at "postmaster@suse.de" as the email to complain to [21:09:55] <[Lemmy]> then it checks its stats and sees 99% bounces from that mail address [21:10:01] but it's tracked on the openSUSE side, we can't do much about suse.de address [21:10:09] can we? [21:10:27] but "we" should prevent such bounces [21:10:36] <[Lemmy]> ...talk to whoever can do something [21:10:52] * jdsn sets up a meeting with me [21:11:05] the RIPE abuse contact for the network range is mikelis-something. [21:11:06] <[Lemmy]> i'm pretty sure there are still channels of communication between o.o and suse.[de|com] [21:11:49] Mikelis Druvetis [21:11:51] <[Lemmy]> next time i run a spam analysis i can see if i can find where spamcop actually gets the postmaster@suse.de from [21:11:52] yes, we still have transports to o.o. mailservers [21:11:59] Redtigra: thanks, exactly. [21:12:24] <[Lemmy]> jsdn: i wasnt talking about smtp between hosts - i meant verbal or other communication between people [21:12:33] [Lemmy]: if you can provide details, please tell me [21:12:38] <[Lemmy]> will do [21:12:54] yes, I already setup a meeting with myself :) [21:13:04] jdsn: have fun :-) [21:13:31] <[Lemmy]> i **think** spamcop uses the abuse.net database to find whom to talk to [21:13:57] postmaster@suse is not a bad start and obviously it should not be bouncing anything. [21:13:58] <[Lemmy]> and since the MX in question was mx2.suse.de it looks up suse.de abuse info there, which returns "hostmaster@suse.de" [21:14:08] <[Lemmy]> https://www.abuse.net/lookup.phtml?domain=suse.de [21:14:45] hostmaster is almost certainly from the domain SOA record [21:15:51] <[Lemmy]> here is an old report [21:15:53] anyway, we are digressing - it is a technical topic, and maybe not really to do with openSUSE any more either? [21:15:53] <[Lemmy]> https://www.spamcop.net/sc?id=z6647250660z47c457495a92fb8dff338b6a9fc00858z#report [21:16:05] agree [21:16:14] <[Lemmy]> it gets the "hostmaster@suse.de" from the ip range [21:16:29] <[Lemmy]> and that address doesnt actually work, or the likes [21:16:56] [Lemmy]: from RIPE ? [21:17:24] <[Lemmy]> all it says is "from whois records" [21:17:29] <[Lemmy]> so i'm guessing ripe [21:18:28] well, this is what I meant to show when I suggested we discuss spam/virus filter fine-tuning. it can take hours to pursue. [21:18:42] it does not mean we shouldn't, just that it takes up time. [21:19:56] <[Lemmy]> huh. looks like spamcop itself isn't doing the right things either - according to whois the abuse mails should actually go to that mikelis guy, not to hostmaster@ [21:20:14] * klein need to go, ciao [21:20:32] cya [21:20:37] [Lemmy]: yeah, hostmaster is weird, it really sounds like it came from an SOA record. [21:20:40] * jdsn leaves as well - N8 [21:21:13] <[Lemmy]> oh whoever said voice meetings instead of irc, how do you keep a written transcript? [21:21:25] <[Lemmy]> just had to throw that in XD [21:21:41] yeah, that's part of the next topic ;-) [21:21:49] we don't keep a written transcript of those meetings either [21:22:00] although it would be easier ;) [21:22:05] I'm done with the spam/virus issue - but always happy to dicsuss. [21:22:47] lcp: which meetings do you mean? [21:23:10] <[Lemmy]> ... https://monitor.opensuse.org/heroes/%23opensuse-admin.2020-09-01.log ? [21:23:29] [Lemmy]: exactly :-) [21:23:30] well, that's channel logs [21:23:31] <[Lemmy]> how do you do that with a voice chat? feed it all to some google service? [21:23:48] <[Lemmy]> that is what i mean, have a meeting in IRC and its easy to have logs [21:23:50] maybe self hosted mozilla voice? [21:23:59] <[Lemmy]> instead of someone learning shorthand XD [21:24:19] for voice meetings, the typical way is to find someone who writes a summary / meeting minutes [21:24:59] either always the same person (no, I don't volunteer ;-) or round-robin to distribute the load [21:25:38] I wonder if the main advantage with voice or video meeting is more adherence to the agenda, to one's turn ? [21:26:15] you can ignore the agenda independent of the medium used ;-) [21:26:27] Voice has another issue: I guess many people don't have English as first language, it is easy for us to miss words or even phrases. Some cases a particular person can not be understood by another person. [21:26:59] it's just easier to know if somebody has anything to say if they keep saying things, and not waiting for the next message ;) [21:27:45] robin_listas: that's a valid concern - for the first two meetings or so. I can tell you from own experience that video meetings are a great way to improve your english skills ;-) [21:28:51] cboltz: yes, it does take some discipline - which I think is also a good thing. [21:29:14] the reason why the idea of video meetings came up was the GDPR meeting last month. It obviously had some "dry" topics, but some people called it "more fun" than an IRC meeting at the end [21:29:36] and that's why I brought this idea up - why should we only have more fun in the GDPR meetings? ;-) [21:30:12] Dress code :-p [21:30:30] I'd say "you have to be dressed" [21:30:42] I'm for doing voice/video calls,. but I accept it would change the format. [21:30:52] cboltz: upper body at least [21:30:53] well, at least the part that's visible for your camera ;-) [21:31:11] <[Lemmy]> guys [21:31:35] <[Lemmy]> somethng like zoom? ugh [21:31:47] we have meet2.opensuse.org :-) [21:31:53] how about a trial run, we have meet.o.o ? if it doesn't work, we can always revert to irc [21:31:58] <[Lemmy]> thats jitsi, isnt it? [21:32:03] yes [21:32:09] [Lemmy]: yup [21:32:23] <[Lemmy]> do you by any chance know how many ppl the biggest meeting on there had? [21:32:37] I can't wait for meet3-9 [21:32:40] we were about 10 in the gdpr meeting [21:32:45] <[Lemmy]> i heard somewhere jitsi doesnt deal well with large groups but the rumor didnt put a number to it [21:33:29] I head (in Chaosradio some months ago) that it mostly depends on the server hardware [21:33:35] <[Lemmy]> it *is* easy enough to set up - i had one up and running including VoIP dialin in about half an hour [21:33:46] <[Lemmy]> docker is nice like that [21:34:03] they said something like "if you apply the BBB hardware requirements to your jitsi server, then all is fine" ;-) [21:34:48] we can always do 2 jitsi calls, and then screenshare the 2 calls with each other [21:35:02] <[Lemmy]> ...via zoom XD [21:35:18] this also scales because you can make it 3 jitsi calls, or 4 jitsi calls [21:35:19] <[Lemmy]> plus a live broadcast on twitch [21:35:20] (the hardware requirements stated by jitsi are lower, and that might be the reason for the rumors you heard about problems with large groups) [21:35:35] <[Lemmy]> could very well be [21:35:56] hehe, I'll be very lucky if we get enough people into the heroes meeting to break jitsi ;-) [21:37:11] cboltz: I will personally fund the hardware upgrade ..... [21:37:37] :-) [21:38:09] so - any objections against doing the next heroes meeting as video call on meet2.o.o? [21:38:20] thumbs up [21:38:32] I'd say we should give it a try, and then decide what we like more [21:38:38] with or without video, by the way. [21:39:33] right, audio-only is also possible [21:39:47] i'm shy [21:40:23] I'm more audio-shy than video-shy tbh [21:40:29] <[Lemmy]> well [21:40:53] <[Lemmy]> if this had been audio you'd be all treated to my wife slaughering large mobs in some MMO [21:41:26] [Lemmy]: and my son .... lots of swearing in swiss grman [21:41:39] What about people wanting to listen, not talk? [21:41:43] <[Lemmy]> there it is - the perfect reason to stick with typing [21:41:46] Or see? [21:42:06] we need to set up a stream >:D [21:42:32] robin_listas: you can always disable your camera nd/or microphone [21:42:34] <[Lemmy]> ok there's always the option to quickly switch to jitsi if there is actually something to show on video or screenshare [21:42:55] <[Lemmy]> i dont even *have* a camera on my main 'puter [21:43:35] quickly switch .... :-) [21:43:55] Same here. My main machine has a camera that never worked. I don't know if the microphone works, either, or headphone. I'd have to use the laptop, with a much smaller display to see people. [21:44:17] it is obvious depends on people - just listening is certainly fine, just like now. I vote with cboltz, it's worth a try. [21:45:21] <[Lemmy]> since i'm no hero i dont vote [21:45:40] yeah, it doesn't hurt to try [21:47:01] how about we schedule the october meeting with jitsi, unless we hear serious complaints? [21:47:57] agreed [21:49:35] I will instruct my son to play tetris [21:50:10] agreed [21:50:45] On occasion I have connected while on the road. Impossible with audio/video. For a test, sure, no problem. [21:51:00] <[Lemmy]> gah now i have the original tetris tune playing in my head on repeat [21:51:06] <[Lemmy]> thank you so much [21:53:07] [Lemmy]: my please :-) [21:53:11] pleasure [21:56:32] <[Lemmy]> i'm outta here - need to find my old gameboy and some batteries [21:56:49] have fun ;-) [21:58:37] does someone have another topic? [22:00:20] I'm done - thanks for being here everyone [22:02:27] thanks everybody for joining the meeting!