[14:46:19] Hello everybody! :) [14:50:01] hi [14:50:35] Who wants to log our meeting? [14:51:42] I'm logging [14:51:58] Cool. :) [14:56:48] I knew that the meeting was postponed because of FOSDEM and now I have some visitors. But, I will check the logs, of course [14:57:36] IonutVan_: some of us are at fosdem [14:57:43] come to the ownCloud booth ;-) [14:57:50] You can read all here, too. That will be enough for you. [14:58:10] cboltz, I am not there :) [14:58:23] We are using the ownCloud wifi at FOSDEM. :) [14:58:25] your fault ;-) [14:58:28] Ada_Lovelace, I will have a look from time to time :) [14:58:45] cboltz, :P [15:00:19] Let's start the meeting. :) [15:00:53] We saw new attacks on our infrastructure. [15:01:09] Look to news.opensuse.org. [15:01:48] *** Joins: orangecms (~cyrevolt@2001:67c:1810:f055:410:62c6:1e9b:8295) [15:01:56] helloooo [15:02:35] Theo? What do you say? [15:03:09] regarding the attack? [15:03:15] Yes [15:03:40] Ada_Lovelace, there is a vulnerability in wordpress 4.7.1. https://blog.sucuri.net/2017/02/content-injection-vulnerability-wordpress-rest-api.html [15:03:45] the issue has been raised to MF-IT to update wordpress and restore the content, and make sure nothing else was touched [15:03:58] also notified my managed christian [15:04:12] meanwhile, if you can find somebody at fosdem that has proper access to restore the content do it please [15:04:18] Ok. We'll recommend Doug to change the password, too. [15:04:24] I asked for access as well [15:04:41] the password doesn't have to be changed [15:04:45] Who is admin for wordpress? [15:05:00] I don't know [15:05:12] I'm trying to figure out [15:05:21] I'll ask Doug. [15:05:41] Anybody gave him access.... [15:06:33] what about a kind of list with: service -> responsible person? I think something like that could be useful [15:07:34] IonutVan_: already wip https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/List_of_machines [15:08:04] *** Joins: lars_ (bc69a4a6@gateway/web/freenode/ip.188.105.164.166) [15:08:13] but news.o.o along with the rest of the services that are hosted in provo have people from MF-IT as responsible, and I am not sure who is supposed to be our opensuse admin from MF-IT provo now [15:08:55] @Lars: Do you know wordpress admins? [15:09:12] Ada_Lovelace: not really, sorry. [15:09:37] ...and sorry for being late, but my pidgin does not want to log in :-/ [15:09:41] Ok. My next step after the meeting: Asking Doug who gave him access. [15:10:00] Do we have a bot - or someone who is keeping a log ? [15:10:01] Ada_Lovelace: the guy that gave him access could be whoever has appropriate perms to do that on the wordpress instance [15:10:02] We speak about the spam on news.opensuse.org [15:10:15] doesn't mean that the same guy has appropriate perms to udpate wordpress [15:10:19] lars_: I am logging [15:10:25] Yes. And the person has admin access... [15:10:27] thanks! [15:10:42] maybe not, this is what I am trying to explain [15:10:50] ok [15:10:57] the interesting detail is that news.o.o seems to get wordpress updates - just not fast enough :-/ [15:11:06] feel free to ask douglas who gave him access, but the person he might point you out doesn't necessarily mean he is the guy we are searching for [15:11:17] you mean that "Hacked By MuhmadEmad" entry? [15:11:23] yes [15:11:32] ah, ok [15:12:05] *** Joins: mrz__ (~mrz@122.177.219.203) [15:12:14] IMHO the problem is the same as with all the other services hosted "outside" Nuremberg: we do not really have access nor do we really know who takes care, right? [15:12:21] We need a wordpress update very fast. [15:12:42] well: we do not only need an update, we need someone who can shut down the instance and investigate the issue [15:12:44] UI access as a admin should be enough [15:12:59] nope it is not [15:13:02] otherwise nobody will know what else might be affected [15:13:14] * cboltz considers switching to S9Y, which is "boring" on the security side (maybe one update per year) [15:13:28] (but obviously, that's a long-term goal, nothing for today) [15:13:48] *** Quits: mrz_ (~mrz@122.177.196.36) (Ping timeout: 240 seconds) [15:14:07] well: I'm always thinking why we do not have the "Automatic-Update" feature enabled for all the wordpress instances, if the admins do not really care [15:14:44] but I'm also not sure if the problem was not simply the password rom Doug - so without further investigation, I doubt that everything we want or do is senseless [15:14:51] it is not the password [15:14:59] so I will sum up because you are missing the details [15:15:06] probably yes, right [15:15:16] there is a vuln on 4.7.1, fixed on 4.7.2 regarding API [15:15:28] I sent a mail to people that might be able to restore the post [15:15:32] tampakrap: do you know if some one escalated the issue already ? [15:15:48] and also raised a ticket to MF-IT, and pointed it to cmueller already so he can mark it as urgent [15:15:55] ok, thanks! [15:15:58] you have a copy of that in ops-services@s.d [15:16:30] so what we need to do is: 1) more people with admin access on the UI (already volunteered for that) 2) figure out who is our provo opensuse contact [15:16:40] what does urgent mean in provo terms? *SCNR* [15:16:43] 3) shut down the instance and check what has been compromised [15:17:02] funny: lizards.opensuse.org seems currently not to be affected [15:17:18] 4) restore the posts [15:17:24] 5) update wordpress and bring it back online [15:17:26] that's all [15:17:35] tampakrap: ui access will not really help if the crap is on the system already ;-) [15:18:00] I know but it is an improvement [15:18:07] because root access I know we can't get [15:18:23] and not only for this case, but for the future as well [15:18:23] That was the reason for me to speak with Doug additionally.... [15:19:11] tampakrap: no, I doubt that we will get root access... [15:19:28] my point exactly [15:19:50] looks like news.o.o is the next thing to migrate to the new infra... [15:19:59] even if they can set up wordpress to run with a dedicated user and give me access to this user I would be happy [15:20:08] so I can update wordpress myself instead of waiting for them [15:20:18] but I'm pretty sure any ssh access to that machine is restricted [15:21:16] tampakrap: well, the plan was to migrate everything openSUSE related to the Provo cloud asap [15:21:25] that included also all the PHP-stuff [15:21:33] What is the status? ;) [15:22:01] but as it looks like we get more and more problems, I think we need to escalate it further and check for alternatives [15:22:34] regarding opensuse cloud, last thing I heard is that it was supposed to be ready on friday, but on friday I was sick and didn't get the final status [15:22:41] Ada_Lovelace: the status is a bit better than a few weeks ago - but still not satisfying [15:23:07] what's the problem this time? [15:23:15] tampakrap: if it would not be so sad, I would say: did you ask which Friday ? ;-) [15:23:48] cboltz: you should ask gschlotter for details [15:24:07] We could also ask for sponsoring at DigitalOcean, they are quite open and approachable [15:24:09] yeah let's not make assumptions, we can have a concrete status from gschlotter on monday [15:24:19] cboltz: but IMHO the access to the storage was the last problem - before the connection got lost completely [15:24:41] but that's all I can tell from a far view point [15:24:49] nice[tm] [15:24:54] orangecms: good idea? Do you have any contacts? [15:25:05] gschlotter: do you know more details? [15:25:30] he is not here [15:26:24] Well, I could approach Allan Jude, who's running the BSDNow podcast, and that is sponsored by DO [15:26:37] He's at the FreeBSD stand :-) [15:26:58] another option would be to ask Hetzner if they can sponsor some servers [15:28:04] noris networks is the sponsor of debian. I don't believe they support openSUSE.... [15:28:32] heinlein is open for us and has got his own ISP. [15:29:16] But he isn't at FOSDEM. [15:29:28] It would also be a cool thing if DigitalOcean could be bumped to offer openSUSE images in general. For now they have Ubuntu, FreeBSD, Fedora, Debian, CoreOS and CentOS [15:29:49] cool ideas! [15:29:54] Should I ask Peer? [15:30:07] I would say: go for it! for all ideas [15:30:17] better to have too much alternatives in the end than nothing ;-) [15:30:45] Ada_Lovelace: ..and you probably want to get the openSUSE board with you [15:31:06] He can become our rack sponsor instead of marketing sponsor. :) [15:31:07] as I think we need to push the Provo guys to get things done asap [15:31:40] this time please _really_ take a video of it ;-) [15:32:23] cboltz: I think "50 shades of Provo" might not be a good title :-) [15:32:45] why not? ;-) [15:33:00] oh, wait - make it "100 shades of Provo" ;-) [15:34:21] do we have some other topics ? [15:34:28] Absolutely, I'll go over there later and poke Allan :D [15:35:28] *** Joins: kl_eisbaer1 (~kl_eisbae@dslb-188-105-164-166.188.105.pools.vodafone-ip.de) [15:35:35] I'll write tomorrow (after my last test in Computer Science this semester). [15:36:23] so we do not have other topics than the hacked news.o.o ? [15:36:41] we have ;-) [15:36:54] do you prefer a nice one or a not-so-nice one first? [15:37:09] after the first topic, I would prefer a nice one :-) [15:37:10] No.No. [15:37:11] Start with the bad ones :D [15:37:21] *seufz* [15:37:24] ok [15:37:28] * orangecms sighs [15:37:39] status of progress [15:38:08] we have lots of bitrotting tickets [15:38:23] and lots of them even block external contributors [15:38:28] Who wants to create all the OBS repositories? [15:38:55] Who wants to add mirrors or fixes issues there? [15:38:56] Ada_Lovelace: that area will be covered by Darix in a few days [15:39:07] define "a few" ;-) [15:39:12] Ada_Lovelace: ...and mirrors was a topic for mcaj and gschlotter [15:39:13] What is "in a few days"? [15:39:25] Ada_Lovelace: once he feels better, I would say [15:39:38] Should we assign it all to them? ;) [15:40:43] Ada_Lovelace: I would say yes [15:41:00] Ok. We'll do. :) [15:41:04] Ada_Lovelace: and use the "Due date" [15:41:18] How long do they have time? [15:41:34] Ada_Lovelace: as they are not here, I would say we need to think for them [15:41:39] what about 2 weeks ? [15:41:40] put me there as well [15:42:25] We are happy with 2 weeks. We would have a really empty queue if they would fix all in this time. :) [15:42:26] should we think about an "obs" and "mirror" group ? [15:42:40] Ok. We'll add you. [15:42:50] we have both [15:42:56] group and category [15:43:18] I would actually kill groups and keep categories only [15:43:20] But admin groups. And Adrian asked why he got these issues. [15:43:40] I don't follow [15:43:45] He doesn't create OBS repositories. [15:43:54] true, darix does [15:44:19] Adrian is in the OBS admin group and doesn't fix "Create repository" issues. [15:44:30] BTW: is there a way to see who is "behind" a group in progress? (as in: who gets mailed?) [15:44:33] correct, don't assign them to obs group but to darix [15:44:41] cboltz: yes at the admin settings [15:44:48] you want a link? [15:44:52] cboltz: IMHO only progress admins can have a look who is in which group [15:45:05] https://progress.opensuse.org/groups [15:45:19] that goes 403 for me [15:45:24] This looks interesting, regaring news.o.o http://www.zone-h.org/archive/notifier=muhmademad?zh=1 [15:46:00] Ada_Lovelace: so we have assigned all mirror and OBS tickets. Are there more ? [15:46:14] Yes. [15:46:53] there are 5 pages with various tickets that don't fit into any group [15:46:59] (or I was too lazy to sort them) [15:47:13] so please check them and assign them to the right person [15:47:18] (and set the due date ;-) [15:47:31] Apparently someone has simply automated crawling the web for WP < 4.7.2 instances and injects her defacements automatically. [15:47:37] We'll have fun with 2 weeks. ;) [15:48:27] anything more? should we move? [15:48:40] next topic: wiki ideas [15:50:18] We had a discusion on our wiki mailing list because of too many tools for documentation. I found the github extension for mediawiki. So we can get upstream docu from github. All will be loaded from there at the timestamp of loading the wiki page. :) [15:50:56] We will have the updated documentation in the wiki you can ever have. ^^ [15:51:01] for an example, see http://paste.opensuse.org/68489949 [15:51:16] (just a random README.md loaded into the test wiki on my laptop) [15:51:24] We tested it today. :) [15:51:34] I don't follow sorry [15:51:37] at the wiki side, it's just {{#github:README.md|openSUSE/geekodoc|develop}} [15:51:40] bear with me, still sick [15:51:58] tampakrap: think of "iframes on steroids" ;-) [15:52:16] you create a github repository, and then load its README.md into the opensuse wiki? [15:52:25] right [15:52:51] and it gets updated automatically if it changes on github [15:53:00] We don't want to write documentation. We want to automate it with the github extension (documentation of different projects on github). [15:53:02] okay and if I change it on the web? [15:53:10] doesn't get automatically committed on github? [15:53:26] in the wiki, you just need {{#github:README.md|openSUSE/geekodoc|develop}} [15:53:34] so you can't edit it directly in the wiki [15:53:38] ah okay, so you can't touch it [15:54:01] okay and which repos' docs you want to load on the wiki? [15:54:12] or you just want to offer it as a generic feature for the community? [15:54:16] You work in github then.That's easier for our developers, too. [15:54:33] we'll offer it as feature for everybody who wants to use it [15:54:38] You can add above/below if you want to state errata or have more info etc [15:54:47] The best thing would be contributing to upstream imho [15:54:59] We want to represent it at the next oSC. [15:55:22] okay [15:55:39] assuming there are no security concerns for this plugin, +1 from me [15:55:55] Thanks! We'll try it. [15:56:55] okay anything further to discuss on it, or it was just to inform us? [15:57:23] We wanted to get your feedback and your SUSE view. [15:58:06] there's no SUSE view, we are all volunteers here :) [15:58:09] next topic? [15:58:13] You know many SOSE developers who write documentations. Do they prefer github? [15:58:44] there are quite of suse/opensuse projects on github [15:59:12] what everyone prefers is not that relevant actually, there are reasons to use a web wiki or a git based documentation based on various factors [15:59:18] Yes. That's it! [16:00:02] so next topic? [16:00:24] tampakrap, should everyone you created an account for on gitlab.o.o have received some individual inviation email? [16:00:35] It seems we have all finished. [16:00:40] so my turn [16:00:59] as orangecms said, gitlab.o.o is on the internet now, along with the salt repository [16:01:30] relevant people should have gotten a mail with their account, and also should have been granted appropriate permissions to see the salt repository [16:01:38] if you didn't get one of those two, let me know please [16:01:58] * orangecms raises a hand [16:02:14] also, I will create two-three VMs for testing purposes so we can experiment [16:02:14] tampakrap: I'm not sure, but I guess if I can not log in any more, I can simply use the "password forgotten" link, right? [16:02:31] orangecms: what's your connect profile please? [16:02:39] lars_: yes and if it doesn't work let me know please [16:02:46] ok, thanks [16:03:04] It's orangecms [16:03:31] in order to check out the repo you need to put in your .ssh/config something that cboltz will tell you, I don't have it now handy sorry :) [16:04:03] I also sent mail to a few opensuse guys that are quite experienced with salt, one of them joined us already [16:04:21] Yup, I've done that already [16:04:37] the SSH config I mean :) [16:04:45] regarding the salt code, you do NOT have access to the production branch (yet) [16:04:55] It's here https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/Salt_repository_layout [16:05:05] the details for SSH are in the admin wiki, and a link to it is on the heroes ML [16:05:12] so you will need to file merge requests and one of me, darix kl_eisbaer and gschlotter should be able to merge it [16:05:28] It's fine, I prefer having a review anyway [16:05:40] brb [16:06:06] for questions regarding ops/structure/whatever feel free to contact me [16:06:13] questions? [16:06:57] I take that as a no [16:06:59] FYI - I'll probably ping you in the next days to get the wiki salt files into the repo [16:07:08] okay [16:07:12] tampakrap: I know it's in the repo, but which hosts and which formulas (or files) are under Salt control now? Do you have some percentages= [16:07:25] s/=/? [16:07:30] lars_: this brings me to my last topic :) [16:07:47] although already said it before, just so that everybody is aware [16:08:03] I have started a wiki page with inventory regarding all of the opensuse hosts https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/List_of_machines [16:08:15] ...and also interesting to me is: https://progress.opensuse.org/issues/16126 [16:08:47] it has the hosts fqdn, cnames, admin contacts, to which salt environment they belong and to which virt cluster [16:09:01] so I started with the machines that are under the opensuse.org salt master, minnie.opensuse.org [16:09:11] as it looks like we need to make sure that we know which packages are currently comming from other repositories before we disable the non-standard ones [16:09:22] and I will continue with the opensuse.org or mixed suse/opensuse hosts that are controlled by SUSE-IT [16:09:28] and the hsots that are in provo [16:09:41] FYI: doug said that henne created his account for news.o.o [16:09:45] tampakrap: this list seems to be handcrafted [16:09:55] yes still wip [16:09:57] tampakrap: isn't there a way to get it automated via Salt ? [16:10:11] otherwise I fear that this table will get outated very soon [16:10:16] it is, but the information is in salt first [16:11:06] tampakrap: ok, so my question would be: can we agree on using only Salt to host this information? [16:11:26] having duplicate information somewhere is always the starting point of nightmares [16:11:32] regarding your repos ticket, I replied there already, I took care of the packages' being in the openSUSE:infrastructure repo, so please be more specific on what is missing so I can fix it (I did like 3 hosts only iirc) [16:11:57] to do this, we need to agree on how to have that info in salt first (documented) [16:12:01] ...and if we need the information about the hosts and their admins in Salt anyway, I do not really see why we need to maintain a manual list elsewhere [16:12:10] then I can create the script that exports them from salt and puts them into the wiki [16:12:29] tampakrap: I did not tell you which hosts I mean because this would be the easy fix ;-) [16:12:31] and yes I really support this idea [16:12:38] lars_: I completely agree about storing this in salt, but _for now_ the list in the wiki is better than nothing [16:12:55] cboltz: putting the info into salt is much less effort than the wiki page [16:12:56] tampakrap: we should run a script that prints out all the packages NOT from our standard repos [16:13:39] tampakrap: if it's currently more effort to maintain the list in the wiki, please stop doing it and concentrate on the real work :-) [16:14:08] once we have time, we can write a script to update a wiki page automatically - if this is still needed [16:14:37] agreed [16:15:15] cboltz: and having an updated list in the wiki is a good starting point for failures, so even if I can understand your wish (especially as wiki admin ;-), I would love to NOT have it [16:15:28] s/updated/outdated/ [16:17:16] for the package issue, here is a very quick hack: [16:17:19] for pkg in $(rpmqpack); do rpm -q --qf "%{NAME} %{DISTURL}\n" $pkg | grep -v "openSUSE:Leap:42.2"; done [16:18:24] or maybe better search for the DISTRIBUTION string, which should not match "openSUSE Leap 42.2" [16:18:57] something like this (together with the list in /etc/nagios/check_zypper-ignores.txt) should bring us on the right track [16:19:27] I guess especially the check_zypper-ignores.txt file should indeed be very helpfull for a first look on a machine [16:19:43] because there I listed all the unmaintained packages anyway [16:19:53] I also have the code ready to install a bunch of packages on salt based on roles (also for common packages) [16:19:56] Theo: You should lolgin into the atreju cluster and shutdown the vm with wordpress. [16:19:58] so it's just a question if all those packages in that file are still installed [16:20:24] Ada_Lovelace: we do not have access to those machines - they are running in Provo, not in NUE [16:20:25] there will be able also the opportunity to select from which repo you want the package installed [16:20:49] then the zypper-ignores.txt can be generated from salt as well [16:20:51] via UI, too? [16:21:01] tampakrap: that sounds great ! [16:21:21] Ada_Lovelace: not at all [16:24:01] I have another topic regarding naming pattern, but I'll skip it for the meeting and probably discuss it over mail [16:24:12] so if there is nothing else to discuss let's close the meeting please [16:24:20] as I'm sick and want to go back to bed [16:24:21] tampakrap: can you sent me the ticket ID for the hacked wordpress , please? [16:24:27] nothing else [16:24:50] lars_: didn't get an incident response yet, do they create them manually? [16:24:59] tampakrap: please do not ask ;-) [16:25:25] tampakrap: JFYI: I added Mike and Roland into the loop [16:25:28] okay meeting over then, next one 5th of march [16:25:40] good [16:25:43] but now to something completely different - if you agree ;-) [16:25:44] good [16:25:59] https://demo.cachethq.io/ [16:25:59] close the log ;) [16:26:16] I was thinking about the status page that I wanted to create [16:26:55] ...and my current assumption (if I look on our "always red" monitoring page): if we just use the output from our monitoring as status page, that will not really be what our users want to see [16:27:07] even if we use the business process add-on [16:27:33] So what do your think about the idea to use the software "Cachet" https://github.com/CachetHQ/Cachet instead ? [16:27:52] there we can create Categories (like OBS, Email, Wiki) and add information there [16:28:00] information from humans like us [16:28:11] or information from ticket or monitoring systems [16:28:35] the only problem I have at the moment. Cachet can be patched to accept LDAP accounts - but not more [16:29:07] and as long as we do not have the FreeIPA solution ready, I'm unsure how to proceed [16:29:31] because I do not want to open it up for everyone inside the community how might have a wiki account to create incidents there [16:29:59] instead again just allowing us access [16:30:16] this might be the same number of people who now has access to the github pages [16:30:48] so I'm a bit blocked by now by the availability of the FreeIPA solution :-( [16:31:06] ....otherwise, it might be a good time now to think if Cachet would make sense as status page ? [16:31:14] What do YOU think? [16:31:29] We have to close the session. The FOSDEM is closed in half an hour. [16:31:37] ok [16:31:54] Just write me an email ;-) [16:32:07] ....and sent some pictures from FOSDEM :-) [16:32:13] I'll add you on CC in the mail to Peer. :) [16:32:51] it seems that the spam was removed from news.o.o. https://news.opensuse.org/ [16:33:06] lars_: no pictures - you'll need to come to FOSDEM next year to see it ;-) [16:33:23] och, menno! ;-) [16:33:46] Doug has got the information. :) [16:34:34] Bye!