Project

General

Profile

Actions

tickets #157798

closed

Migrate FreeIPA consumers to Kanidm

Added by crameleon about 1 month ago. Updated 18 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
FreeIPA/Kanidm
Target version:
-
Start date:
2024-03-23
Due date:
% Done:

100%

Estimated time:

Description

This tracks switching all Heroes services from FreeIPA to Kanidm authentication.
Where possible, LDAP should be replaced with Oauth2/OIDC as part of the process. PKCE should be used where supported.


Checklist

  • GitLab
  • Grafana
  • Matomo
  • Unix clients

Related issues 1 (0 open1 closed)

Related to openSUSE admin - tickets #154984: beans.opensuse.org login issuesClosedcrameleon2024-02-06

Actions
Actions #1

Updated by crameleon about 1 month ago

  • Status changed from New to In Progress
  • Private changed from Yes to No
Actions #2

Updated by crameleon about 1 month ago

  • Checklist item GitLab set to Done

GitLab done. As part of the change, GitLab administrators are now managed using the "gitlab-admins" group in Kanidm.

Actions #3

Updated by crameleon about 1 month ago · Edited

  • Checklist item Grafana set to Done

Grafana done. As part of the change, Grafana administrators are now managed through the "grafana-admins" group, and editors through the "grafana-editors" group in Kanidm.

Whereas GitLab migrated users from LDAP to Oauth2 "under the hood", Grafana was more involved.
Using oauth2_prefer_short_username: true in the Oauth2 client for Grafana in Kanidm, Grafana would notice the existing users, but try to create new ones "on top", failing with unique database constraints.
One solution would have been to use SPN based usernames (the default with Kanidm if oauth2_prefer_short_username is not set) and just delete the old, stale, short name based ones, given them not containing much data anyways.
But out of curiosity, I worked out how to migrate the existing short name based users in the Grafana database.

Example based on my own account:

First, query the existing data from the user and user_auth tables:

sqlite> SELECT * FROM user;
...
7|0|crameleon|crameleon|||xxx|xxx||1|1|0||2023-11-26 20:04:57|2023-11-26 20:04:57|1|2024-03-23 14:09:39|0|0

sqlite> SELECT * FROM user_auth;
...
7|7|ldap|uid=crameleon,cn=users,cn=compat,dc=infra,dc=opensuse,dc=org|2024-03-21 20:25:21|||||

Note the id of my account is 7.

  • Change the login field from the short name to the primary email address (the first mail field reported by kanidm person get crameleon):
sqlite> UPDATE user set login = 'crameleon@opensuse.org' WHERE id = 7;
  • Change the auth_module field:
sqlite> UPDATE user_auth SET auth_module = 'oauth_generic_oauth' WHERE user_id = 7;
  • Change LDAP DN to UUID (the uuid field reported by kanidm person get crameleon):
sqlite> UPDATE user_auth SET auth_id = 'c9b48c02-e8ff-11ec-9769-b47042ddcecb' WHERE user_id = 7;

Now authenticating with Oauth2/Kanidm gets me into my old/existing account.

For larger Grafana installations this should probably be scripted.
In our case a mere 5 unique users ever logged in to the instance, hence I just did UPDATE user_auth SET auth_module = 'oauth_generic_oauth' WHERE auth_module = 'ldap'; and then updated login and auth_id manually for every user.

Actions #4

Updated by crameleon about 1 month ago

  • Category changed from Core services and virtual infrastructure to FreeIPA/Kanidm
  • Assignee changed from opensuse-admin to crameleon
Actions #5

Updated by crameleon about 1 month ago

Actions #6

Updated by crameleon about 1 month ago

  • Checklist item Matomo set to Done

Since the Matomo authentication is currently broken anyways, and the lack of logging in the application making both repairing the existing issues as well as switching it to the new system unfeasible, I will mark it as complete without action.
If we repair logging in Matomo in the future, I can implement https://plugins.matomo.org/LoginOIDC there. Since it's not clear when it will happen, it shouldn't be blocking the migration.

Actions #7

Updated by crameleon about 1 month ago

All minions in Prague have been switched to Kanidm.
Major regression due to missing to test on Tumbleweed was the removal of systemd which I now patched via https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/1627.

Actions #8

Updated by crameleon about 1 month ago

One originally minor issue I found is that sometimes it takes ~2 attempts to SSH into a machine, I assume it needs to fill the cache or similar? This became less minor when migrating minions in Nuremberg, where fail2ban is employed, and one doesn't get that many attempts.

Another obscurity is with slimhat.infra.opensuse.org, somehow I cannot authenticate:

crameleon@slimhat.infra.opensuse.org: Permission denied (publickey,keyboard-interactive).

    Mar 31 16:19:51 slimhat sshd[30978]: Invalid user crameleon from 2a07:de40:b27e:5001:f797:9eb7:983:bc9f port 49008
    Mar 31 16:19:51 slimhat sshd[30978]: error: PAM: User not known to the underlying authentication module for illegal user crameleon from 2a07:de40:b27e:5001:f797:9eb7:983:bc9f
    Mar 31 16:19:51 slimhat sshd[30978]: Failed keyboard-interactive/pam for invalid user crameleon from 2a07:de40:b27e:5001:f797:9eb7:983:bc9f port 49008 ssh2
    Mar 31 16:19:51 slimhat sshd[30978]: Connection closed by invalid user crameleon 2a07:de40:b27e:5001:f797:9eb7:983:bc9f port 49008 [preauth]

kanidm-unixd and kanidm-unixd-tasks are started normally, and /usr/sbin/kanidm_ssh_authorizedkeys crameleon works as well.

Actions #9

Updated by crameleon about 1 month ago · Edited

# salt slimhat.infra.opensuse.org cmd.run 'id crameleon'
slimhat.infra.opensuse.org:
    id: 'crameleon': no such user

# salt slimhat.infra.opensuse.org cmd.run 'getent passwd crameleon'
slimhat.infra.opensuse.org:
<empty>

Comparing nsswitch.conf against a working machine:

~/georg# diff -bU0 nss*.sorted
--- nss-slimhat.sorted  2024-03-31 16:32:25.215456402 +0000
+++ nss-witch1.sorted   2024-03-31 16:32:19.003253764 +0000
@@ -1 +1 @@
-aliases:    files
+aliases:       files usrfiles
@@ -4 +4 @@
-ethers:             files
+ethers:                files usrfiles
@@ -5,0 +6 @@
+gshadow:       files usrfiles
@@ -11 +12 @@
-protocols:  files
+protocols:     files usrfiles
@@ -13,2 +14,3 @@
-rpc:                files
-services:   files
+rpc:           files usrfiles
+services:      files usrfiles
+shadow:                compat

... and curl -sI https://idm.infra.opensuse.org works.

Actions #10

Updated by crameleon about 1 month ago

Another "odd" client: chip.infra.opensuse.org - here authentication works, but I am always prompted for a passphrase.

Actions #11

Updated by crameleon 29 days ago

  • slimhat resolved by magic, chip was found to be a different issue
  • @firstyear investigates double-attempt issue and raises connection timeout
Actions #12

Updated by crameleon 28 days ago

  • % Done changed from 0 to 70

Patch to raise timeouts applied, all minions in Nuremberg migrated.

Actions #13

Updated by crameleon 28 days ago

  • Checklist item Unix clients set to Done
  • % Done changed from 70 to 100

All minions in Provo migrated as well. Highly unspectacular.

Actions #14

Updated by crameleon 18 days ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF