tickets #157798
closedMigrate FreeIPA consumers to Kanidm
100%
Description
This tracks switching all Heroes services from FreeIPA to Kanidm authentication.
Where possible, LDAP should be replaced with Oauth2/OIDC as part of the process. PKCE should be used where supported.
Updated by crameleon 3 months ago · Edited
- Checklist item Grafana set to Done
Grafana done. As part of the change, Grafana administrators are now managed through the "grafana-admins" group, and editors through the "grafana-editors" group in Kanidm.
Whereas GitLab migrated users from LDAP to Oauth2 "under the hood", Grafana was more involved.
Using oauth2_prefer_short_username: true in the Oauth2 client for Grafana in Kanidm, Grafana would notice the existing users, but try to create new ones "on top", failing with unique database constraints.
One solution would have been to use SPN based usernames (the default with Kanidm if oauth2_prefer_short_username is not set) and just delete the old, stale, short name based ones, given them not containing much data anyways.
But out of curiosity, I worked out how to migrate the existing short name based users in the Grafana database.
Example based on my own account:
First, query the existing data from the user and user_auth tables:
sqlite> SELECT * FROM user;
...
7|0|crameleon|crameleon|||xxx|xxx||1|1|0||2023-11-26 20:04:57|2023-11-26 20:04:57|1|2024-03-23 14:09:39|0|0
sqlite> SELECT * FROM user_auth;
...
7|7|ldap|uid=crameleon,cn=users,cn=compat,dc=infra,dc=opensuse,dc=org|2024-03-21 20:25:21|||||
Note the id of my account is 7.
- Change the
loginfield from the short name to the primary email address (the firstmailfield reported bykanidm person get crameleon):
sqlite> UPDATE user set login = 'crameleon@opensuse.org' WHERE id = 7;
- Change the
auth_modulefield:
sqlite> UPDATE user_auth SET auth_module = 'oauth_generic_oauth' WHERE user_id = 7;
- Change LDAP DN to UUID (the
uuidfield reported bykanidm person get crameleon):
sqlite> UPDATE user_auth SET auth_id = 'c9b48c02-e8ff-11ec-9769-b47042ddcecb' WHERE user_id = 7;
Now authenticating with Oauth2/Kanidm gets me into my old/existing account.
For larger Grafana installations this should probably be scripted.
In our case a mere 5 unique users ever logged in to the instance, hence I just did UPDATE user_auth SET auth_module = 'oauth_generic_oauth' WHERE auth_module = 'ldap'; and then updated login and auth_id manually for every user.
Updated by crameleon 3 months ago
- Related to tickets #154984: beans.opensuse.org login issues added
Updated by crameleon 3 months ago
- Checklist item Matomo set to Done
Since the Matomo authentication is currently broken anyways, and the lack of logging in the application making both repairing the existing issues as well as switching it to the new system unfeasible, I will mark it as complete without action.
If we repair logging in Matomo in the future, I can implement https://plugins.matomo.org/LoginOIDC there. Since it's not clear when it will happen, it shouldn't be blocking the migration.
Updated by crameleon 3 months ago
All minions in Prague have been switched to Kanidm.
Major regression due to missing to test on Tumbleweed was the removal of systemd which I now patched via https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1627.
Updated by crameleon 3 months ago
One originally minor issue I found is that sometimes it takes ~2 attempts to SSH into a machine, I assume it needs to fill the cache or similar? This became less minor when migrating minions in Nuremberg, where fail2ban is employed, and one doesn't get that many attempts.
Another obscurity is with slimhat.infra.opensuse.org, somehow I cannot authenticate:
crameleon@slimhat.infra.opensuse.org: Permission denied (publickey,keyboard-interactive).
Mar 31 16:19:51 slimhat sshd[30978]: Invalid user crameleon from 2a07:de40:b27e:5001:f797:9eb7:983:bc9f port 49008
Mar 31 16:19:51 slimhat sshd[30978]: error: PAM: User not known to the underlying authentication module for illegal user crameleon from 2a07:de40:b27e:5001:f797:9eb7:983:bc9f
Mar 31 16:19:51 slimhat sshd[30978]: Failed keyboard-interactive/pam for invalid user crameleon from 2a07:de40:b27e:5001:f797:9eb7:983:bc9f port 49008 ssh2
Mar 31 16:19:51 slimhat sshd[30978]: Connection closed by invalid user crameleon 2a07:de40:b27e:5001:f797:9eb7:983:bc9f port 49008 [preauth]
kanidm-unixd and kanidm-unixd-tasks are started normally, and /usr/sbin/kanidm_ssh_authorizedkeys crameleon works as well.
Updated by crameleon 3 months ago · Edited
# salt slimhat.infra.opensuse.org cmd.run 'id crameleon'
slimhat.infra.opensuse.org:
id: 'crameleon': no such user
# salt slimhat.infra.opensuse.org cmd.run 'getent passwd crameleon'
slimhat.infra.opensuse.org:
<empty>
Comparing nsswitch.conf against a working machine:
~/georg# diff -bU0 nss*.sorted
--- nss-slimhat.sorted 2024-03-31 16:32:25.215456402 +0000
+++ nss-witch1.sorted 2024-03-31 16:32:19.003253764 +0000
@@ -1 +1 @@
-aliases: files
+aliases: files usrfiles
@@ -4 +4 @@
-ethers: files
+ethers: files usrfiles
@@ -5,0 +6 @@
+gshadow: files usrfiles
@@ -11 +12 @@
-protocols: files
+protocols: files usrfiles
@@ -13,2 +14,3 @@
-rpc: files
-services: files
+rpc: files usrfiles
+services: files usrfiles
+shadow: compat
... and curl -sI https://idm.infra.opensuse.org works.
Updated by crameleon 3 months ago
- slimhat resolved by magic, chip was found to be a different issue
- @firstyear investigates double-attempt issue and raises connection timeout
Updated by crameleon 17 days ago · Edited
- Checklist item Matomo set to Not done
- Status changed from Resolved to In Progress
- % Done changed from 100 to 90
Re-opening, Matomo is needed.
- OIDC plugin packaged and installed.
- Integration with Kanidm configured.
- Mostly works but conflicts with LDAP plugin => wait for Douglas to figure out VPN access, then remove LDAP plugin.
Updated by crameleon 17 days ago
- Blocked by tickets #161948: Heroes VPN access for ddemaio added
Updated by crameleon 17 days ago
- Blocks tickets #161105: Join openSUSE Heroes, request for an account and VPN access added
Updated by crameleon 8 days ago
Removed the LDAP plugin now, but it seems the error which I thought was a conflict of methods between the two plugins still occurs:
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] Uncaught exception: Error: Call to undefined method Piwik\Plugins\Login\Auth::setForceLogin() in /srv/www/matomo/plugins/LoginOIDC/Controller.php:431
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] Stack trace:
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] #0 /srv/www/matomo/plugins/LoginOIDC/Controller.php(310): Piwik\Plugins\LoginOIDC\Controller->signinAndRedirect()
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] #1 [internal function]: Piwik\Plugins\LoginOIDC\Controller->callback()
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] #2 /srv/www/matomo/core/FrontController.php(637): call_user_func_array()
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] #3 /srv/www/matomo/core/FrontController.php(169): Piwik\FrontController->doDispatch()
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] #4 /srv/www/matomo/core/dispatch.php(32): Piwik\FrontController->dispatch()
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] #5 /srv/www/matomo/index.php(25): require_once('...')
Jun 21 13:07:35 matomo matomo[22433]: ERROR Piwik\FrontController[2024-06-21 13:07:35 UTC] [36de7] #6 {main}
Jun 21 13:07:35 matomo matomo[22433]:
Updated by crameleon 8 days ago
Solved via https://github.com/dominik-th/matomo-plugin-LoginOIDC/pull/116 (not sure if this is the "right" way / whether that import was there for some reason I missed - but it works in our case). Will submit the patch to our package.
Updated by crameleon 8 days ago
- Checklist item Matomo set to Done
- Status changed from In Progress to Resolved
- % Done changed from 90 to 100
Matomo done, one patch to improve UX pending via https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1935, and a follow up problem tracked via https://progress.opensuse.org/issues/162722, but no blockers here.