New Redmine system¶

The work to upgrade the current Redmine system here (https://progress.opensuse.org) has started. Achievements so far:

• https://progress-test.opensuse.org/ is up and running with latest 4.2.1 stable version
• Redmine plus dependencies are coming from packages in openSUSE:infrastructure:redmine (not from plain source)
• an old DB dump has been imported and upgraded successfully
• authentication has been configured to use LDAP directly (no need for the unsupported iChain plugin any longer). This needs adjustments for all active Redmine users in the Database. Luckily, just a switch in one DB field in one table (auth_source_id in the users table has to be set from NULL to 3). Question is, if we really should we upgrade all 'affected' 17065 accounts: most of them are SPAM accounts ? - it 'might' not harm, but this is untested so far. People, who want to check out the latest instance, can sent an Email to Lars to get access. Please note: the DB dump is older - and the import and upgrade needs to be done again, once we finalize the setup.
• The new version also allows a log-in via openID provider - which might be an alternative solution (but IMHO not needed, as with direct LDAP support, we can leave the log-in as it is right now.
• The first two plugins have been upgraded (needed, as none of the old plugins support the latest version), installed and activated:
  • Redmine Agile plugin (Light version)
  • Redmine Checklists plugin (Light version)

Both plugins come from https://www.redmineup.com/ - and are crippled with a closed source license for the images and css. That's why we don't have packages for them (yet). Re-packaging by using the old images and patched CSS might be doable, but this is not considered yet. Question: do we really need/want these plugins?

TODO:

• theming - the new version is currently completely unthemed
• upgrade the rest of the current plugins (9 left):
  • Due Date Reminder plugin - Sends notifications about due date (Prio 2)
  • Redmine diary - Diary view for time entries plus some related goodies (Prio 4)
  • Favorite Projects - This is a favorite projects plugin for Redmine (Prio 5)
  • Redmine Force issues to be private - Redmine Force issues to be privat by default (Prio 1)
  • Redmine plugin to post a simple message to the login page which account type is used - Redmine plugin to post a simple message to the login page which account type is used (Prio 5)
  • Redmine Mail Reminder plugin - Issue reminder plugin for Redmine (Prio 2)
  • Redmine plugin views revisions plugin - This plugin tries to solve problem that is caused by inability to monkey-patch views in the Redmine. For details please see http://www.redmine.org/plugins/redmine_plugin_views_revisions for more details (Prio 4)
  • Redmine Tags plugin - Redmine issues tagging support (Prio 3)
  • Reopen issues by mail plugin - Very simple (and a little bit hacky) plugin which adjusts the state of a closed issue after receiving an update by email (Prio 3)

Prio 1-4 plugins are currently planned to get upgraded and working with the new version in the order of their priority. If a plugin can not be upgraded, we need to find another solution.

Prio 5 plugins are currently considered to be obsoleted and completely skipped. If you need a Prio 5 plugin, please speak up - or or hold your peace forever. ;-)

Finally: a list of "plugin owners" might be useful, to see which team requires a specific Redmine Plugin. This might become even more important, if the tradition to release crippled plugins continues. We have two plugins already, that lost some of their features now. If someone pays for them, we are allowed to use the "professional" variant with all features enabled.

New backup system¶

The new backup server is up and running. It holds 3TB space for backups on a dedicated disk. So far, we are still in testing mode. Starting with unencrypted NFS exports for urgent matters...

Borg Backup or restic are currently considered for the final solution.

Experts are needed. This might be an interesting job for newbies...?

Gitlab upgrade¶

Gitlab was upgraded to the latest 14.1 version, fixing some security issues. This upgrade also silently switched from Unicorn to Puma in the background, which resulted in an unexpected downtime.

There is currently an open issue ( #95896 ) about ssh access to repos, which needs further debugging.

Monitoring update¶

Our monitoring checks now 79 machines and 1655 Services. Distribution updates are installed on all of them (beside the known machines with an unsupported OS).

Problem¶

There are still some machines, that are not even in base monitoring. Most of these machines are not running salt, so there is no way to get the needed plugins installed on them.

Question: how to handle "secured" machines?

• So far, most machines in the openSUSE Heroes network allow SSH access from anyone with a FreeIPA account. At least basic monitoring is (or will) be enabled on all of them.
• Some specific machines (LDAP and Login Proxies, Machines handling the internal or external SSL certificates) only allow SSH access from SUSE employees. All of them allow monitoring.
• identification.infra.opensuse.org (#92740) or code.infra.opensuse.org (#94375) are allowing only some Heroes to log in. They are neither in Salt or in Monitoring.

Suggestion:

• have all machines under openSUSE heroes control in Salt
• have all machines under openSUSE heroes control in Monitoring (test machines with basic monitoring)
• document machines, that have special access permissions at least in our admin wiki (even in Gitlab? Where? IMHO adding them in pillars will result in Salt timeouts and errors, so this might not be useful...?).

We might oversee the "special machines" very easy, if they don't pop up in our monitoring. This might result in missing security updates (which is also the reason to include even test machines in our monitoring) or other problems like downtime of services (like authentication issues or outdated certificates).

mirrordb2¶

Is still running PostgreSQL 12. Should we do the upgrade to PostgreSQL 13 again, to benefit from the additional performance?

Who likes to help with a redundant setup in Provo or in another data center?

lrupp wrote:

New Redmine system¶

[...] People, who want to check out the latest instance, can sent an Email to Lars to get access.

not an email, but still: I would like to have an access :)

• The first two plugins have been upgraded (needed, as none of the old plugins support the latest version), installed and activated:
  • Redmine Agile plugin (Light version)
  • Redmine Checklists plugin (Light version)

Both plugins come from https://www.redmineup.com/ - and are crippled with a closed source license for the images and css. That's why we don't have packages for them (yet). Re-packaging by using the old images and patched CSS might be doable, but this is not considered yet. Question: do we really need/want these plugins?

I am sure that some people from the SUSE LSG QE department would like to have the "Redmine Agile Plugin" at least

Finally: a list of "plugin owners" might be useful, to see which team requires a specific Redmine Plugin. This might become even more important, if the tradition to release crippled plugins continues. We have two plugins already, that lost some of their features now. If someone pays for them, we are allowed to use the "professional" variant with all features enabled.

If you need sponsoring for the "Redmine Agile plugin" I think, as stated above, there is interest from SUSE LSG QE department and potential for sponsoring.

Replaced widehat aka rsync.opensuse.org¶

The old widehat hat "just" 19TB space to mirror content from download.opensuse.org. This system has now been replaced by a new one providing 42TB space. This allows to mirror everything which is currently provided by OBS!

Edit (jdsn): To be precise, the old widehat was replaced by stonehat and widehat became a VM on that new host.

The new system also has some resources left to run additional workloads: we might think about moving once of the static content nodes or MX to it.

At the moment, the new system still needs some fine tuning (we want to set it up in a similar way like the systems in Provo or Nuremberg), which includes some changes in the network setup (connection into the infra.opensuse.org network via wireguard, an own, private IP range for machines, etc). These configuration changes might need some time.

The new widehat also lost one of it's public interfaces (which is now acquired by the hypervisor). This needs changes in the publishing (from OBS to widehat) and scanning (from scanner and stage to widehat), which is currently processed.

Matomo¶

Upgraded to latest (4.4.1) version as usual. Working on tickets around Matomo (#90710, #90512, #95673).