Project

General

Profile

communication #95144

2021-08-03 18:00 UTC: openSUSE Heroes meeting August 2021

Added by cboltz 2 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
opensuse-admin
Category:
Event
Target version:
-
Start date:
2021-07-06
Due date:
% Done:

0%

Estimated time:

Description

Where: https://meet.opensuse.org/heroes
When: 2021-08-03 18:00 UTC / 20:00 CEST
Who: The openSUSE Heroes team and everybody else!

Topics
see/use checklist


Checklist

  • Questions and answers from the community
  • status reports about everything
  • review old tickets
  • Drop Redmine plugin: "post a simple message to the login page"
  • Drop Redmine plugin: "Favorite Projects"
  • Help with Backup server/service wanted
  • Gitlab issue needs fixing after security update
  • How to handle "secured" machines?
  • Help with PostgreSQL setup wanted
  • Create an infrastructure survey?

History

#1 Updated by cboltz 2 months ago

  • Private changed from Yes to No

#2 Updated by lrupp about 2 months ago

New Redmine system

The work to upgrade the current Redmine system here (https://progress.opensuse.org) has started. Achievements so far:

  • https://progress-test.opensuse.org/ is up and running with latest 4.2.1 stable version
  • Redmine plus dependencies are coming from packages in openSUSE:infrastructure:redmine (not from plain source)
  • an old DB dump has been imported and upgraded successfully
  • authentication has been configured to use LDAP directly (no need for the unsupported iChain plugin any longer). This needs adjustments for all active Redmine users in the Database. Luckily, just a switch in one DB field in one table (auth_source_id in the users table has to be set from NULL to 3). Question is, if we really should we upgrade all 'affected' 17065 accounts: most of them are SPAM accounts ? - it 'might' not harm, but this is untested so far. People, who want to check out the latest instance, can sent an Email to Lars to get access. Please note: the DB dump is older - and the import and upgrade needs to be done again, once we finalize the setup.
  • The new version also allows a log-in via openID provider - which might be an alternative solution (but IMHO not needed, as with direct LDAP support, we can leave the log-in as it is right now.
  • The first two plugins have been upgraded (needed, as none of the old plugins support the latest version), installed and activated:
    • Redmine Agile plugin (Light version)
    • Redmine Checklists plugin (Light version)

Both plugins come from https://www.redmineup.com/ - and are crippled with a closed source license for the images and css. That's why we don't have packages for them (yet). Re-packaging by using the old images and patched CSS might be doable, but this is not considered yet. Question: do we really need/want these plugins?

TODO:

  • theming - the new version is currently completely unthemed
  • upgrade the rest of the current plugins (9 left):
    • Due Date Reminder plugin - Sends notifications about due date (Prio 2)
    • Redmine diary - Diary view for time entries plus some related goodies (Prio 4)
    • Favorite Projects - This is a favorite projects plugin for Redmine (Prio 5)
    • Redmine Force issues to be private - Redmine Force issues to be privat by default (Prio 1)
    • Redmine plugin to post a simple message to the login page which account type is used - Redmine plugin to post a simple message to the login page which account type is used (Prio 5)
    • Redmine Mail Reminder plugin - Issue reminder plugin for Redmine (Prio 2)
    • Redmine plugin views revisions plugin - This plugin tries to solve problem that is caused by inability to monkey-patch views in the Redmine. For details please see http://www.redmine.org/plugins/redmine_plugin_views_revisions for more details (Prio 4)
    • Redmine Tags plugin - Redmine issues tagging support (Prio 3)
    • Reopen issues by mail plugin - Very simple (and a little bit hacky) plugin which adjusts the state of a closed issue after receiving an update by email (Prio 3)

Prio 1-4 plugins are currently planned to get upgraded and working with the new version in the order of their priority. If a plugin can not be upgraded, we need to find another solution.

Prio 5 plugins are currently considered to be obsoleted and completely skipped. If you need a Prio 5 plugin, please speak up - or or hold your peace forever. ;-)

Finally: a list of "plugin owners" might be useful, to see which team requires a specific Redmine Plugin. This might become even more important, if the tradition to release crippled plugins continues. We have two plugins already, that lost some of their features now. If someone pays for them, we are allowed to use the "professional" variant with all features enabled.

#3 Updated by lrupp about 2 months ago

New backup system

The new backup server is up and running. It holds 3TB space for backups on a dedicated disk. So far, we are still in testing mode. Starting with unencrypted NFS exports for urgent matters...

Borg Backup or restic are currently considered for the final solution.

Experts are needed. This might be an interesting job for newbies...?

#4 Updated by lrupp about 2 months ago

Gitlab upgrade

Gitlab was upgraded to the latest 14.1 version, fixing some security issues. This upgrade also silently switched from Unicorn to Puma in the background, which resulted in an unexpected downtime.

There is currently an open issue ( #95896 ) about ssh access to repos, which needs further debugging.

#5 Updated by lrupp about 2 months ago

Monitoring update

Our monitoring checks now 79 machines and 1655 Services. Distribution updates are installed on all of them (beside the known machines with an unsupported OS).

Problem

There are still some machines, that are not even in base monitoring. Most of these machines are not running salt, so there is no way to get the needed plugins installed on them.

Question: how to handle "secured" machines?

  • So far, most machines in the openSUSE Heroes network allow SSH access from anyone with a FreeIPA account. At least basic monitoring is (or will) be enabled on all of them.
  • Some specific machines (LDAP and Login Proxies, Machines handling the internal or external SSL certificates) only allow SSH access from SUSE employees. All of them allow monitoring.
  • identification.infra.opensuse.org (#92740) or code.infra.opensuse.org (#94375) are allowing only some Heroes to log in. They are neither in Salt or in Monitoring.

Suggestion:

  • have all machines under openSUSE heroes control in Salt
  • have all machines under openSUSE heroes control in Monitoring (test machines with basic monitoring)
  • document machines, that have special access permissions at least in our admin wiki (even in Gitlab? Where? IMHO adding them in pillars will result in Salt timeouts and errors, so this might not be useful...?).

We might oversee the "special machines" very easy, if they don't pop up in our monitoring. This might result in missing security updates (which is also the reason to include even test machines in our monitoring) or other problems like downtime of services (like authentication issues or outdated certificates).

#6 Updated by lrupp about 2 months ago

mirrordb2

Is still running PostgreSQL 12. Should we do the upgrade to PostgreSQL 13 again, to benefit from the additional performance?

Who likes to help with a redundant setup in Provo or in another data center?

#7 Updated by lrupp about 2 months ago

  • Checklist changed from [ ] Questions and answers from the community, [ ] status reports about everything, [ ] review old tickets to [ ] Questions and answers from the community, [ ] status reports about everything, [ ] review old tickets, [ ] Drop Redmine plugin: "post a simple message to the login page", [ ] Drop Redmine plugin: "Favorite Projects", [ ] Help with Backup server/service wanted, [ ] Gitlab issue needs fixing after security update, [ ] How to handle "secured" machines?, [ ] Help with PostgreSQL setup wanted

Shutdown of connect

The machine itself is still up and running, but the frontend proxy is not using it any longer - and redirects to the "deprecated" page instead. Next step: decommission the machine.

The cron job on the two MX servers is currently disabled, to avoid empty user alias files.

Problem

  • We should provide an alternative solution to the membership committee
  • We might want to provide an alternative solution to our users

Membership management ideas

Manage membership via Redmine

  • Create an own (hidden) project here at progress
  • Create an own Email queue, so users can request their membership via Email - and the Email ends up in a Redmine ticket, which can be handled like here in the admin project
  • Use the Redmine API to provide a membership list to other services (like our MX) - to be clarified

Manage membership via Pagure

  • Create an own, project at Pagure
  • allow users to open an issue against this new project
  • let the membership committee handle the issue
  • Use the projects file repository to provide a membership list to other services (like our MX)
  • To be clarified:
    • Does Pagure allow to open tickets via Email?
    • Is it possible to mark specific issues as non-public - only visible to certain users?

Manage membership via RT

  • Create an own queue in our RT test instance
  • Create an own Email queue, so users can request their membership via Email - and the Email ends up in a RT ticket
  • Use the RT API to provide a membership list to other services (like our MX) - to be clarified

Manage membership via wiki

  • Is this really an alternative?

Social project for our community

  • Our community could use our Moodle instance for socializing
  • A new Nextcloud or Owncloud instance, including some plugins (like the share Calendar, Circles, ... might also be possible, but this needs more manpower, resp. volunteers.
  • Maybe we need a survey first, to see what our community is currently missing?

#8 Updated by lrupp about 2 months ago

  • Checklist changed from [ ] Questions and answers from the community, [ ] status reports about everything, [ ] review old tickets, [ ] Drop Redmine plugin: "post a simple message to the login page", [ ] Drop Redmine plugin: "Favorite Projects", [ ] Help with Backup server/service wanted, [ ] Gitlab issue needs fixing after security update, [ ] How to handle "secured" machines?, [ ] Help with PostgreSQL setup wanted to [ ] Questions and answers from the community, [ ] status reports about everything, [ ] review old tickets, [ ] Drop Redmine plugin: "post a simple message to the login page", [ ] Drop Redmine plugin: "Favorite Projects", [ ] Help with Backup server/service wanted, [ ] Gitlab issue needs fixing after security update, [ ] How to handle "secured" machines?, [ ] Help with PostgreSQL setup wanted, [ ] Create an infrastructure survey?

#9 Updated by okurz about 2 months ago

lrupp wrote:

New Redmine system

[...] People, who want to check out the latest instance, can sent an Email to Lars to get access.

not an email, but still: I would like to have an access :)

  • The first two plugins have been upgraded (needed, as none of the old plugins support the latest version), installed and activated:
    • Redmine Agile plugin (Light version)
    • Redmine Checklists plugin (Light version)

Both plugins come from https://www.redmineup.com/ - and are crippled with a closed source license for the images and css. That's why we don't have packages for them (yet). Re-packaging by using the old images and patched CSS might be doable, but this is not considered yet. Question: do we really need/want these plugins?

I am sure that some people from the SUSE LSG QE department would like to have the "Redmine Agile Plugin" at least

Finally: a list of "plugin owners" might be useful, to see which team requires a specific Redmine Plugin. This might become even more important, if the tradition to release crippled plugins continues. We have two plugins already, that lost some of their features now. If someone pays for them, we are allowed to use the "professional" variant with all features enabled.

If you need sponsoring for the "Redmine Agile plugin" I think, as stated above, there is interest from SUSE LSG QE department and potential for sponsoring.

#10 Updated by lrupp about 2 months ago

Replaced widehat aka rsync.opensuse.org

The old widehat hat "just" 19TB space to mirror content from download.opensuse.org. This system has now been replaced by a new one providing 42TB space. This allows to mirror everything which is currently provided by OBS!

Edit (jdsn): To be precise, the old widehat was replaced by stonehat and widehat became a VM on that new host.

The new system also has some resources left to run additional workloads: we might think about moving once of the static content nodes or MX to it.

At the moment, the new system still needs some fine tuning (we want to set it up in a similar way like the systems in Provo or Nuremberg), which includes some changes in the network setup (connection into the infra.opensuse.org network via wireguard, an own, private IP range for machines, etc). These configuration changes might need some time.

The new widehat also lost one of it's public interfaces (which is now acquired by the hypervisor). This needs changes in the publishing (from OBS to widehat) and scanning (from scanner and stage to widehat), which is currently processed.

#11 Updated by lrupp about 2 months ago

Matomo

Upgraded to latest (4.4.1) version as usual. Working on tickets around Matomo (#90710, #90512, #95673).

#12 Updated by jdsn about 1 month ago

  • Checklist set to [x] Questions and answers from the community

#13 Updated by cboltz about 1 month ago

  • Status changed from New to Closed

2021-08-03 heroes meeting

Also available in: Atom PDF