Project

General

Profile

Actions

tickets #173533

open

kanidmd reload failure

Added by crameleon 25 days ago. Updated 23 days ago.

Status:
New
Priority:
Normal
Assignee:
Category:
FreeIPA/Kanidm
Target version:
-
Start date:
2024-12-02
Due date:
% Done:

0%

Estimated time:

Description

Monitoring complains about failing reload after certificate renewals on kani{1,2}. It seems kanidmd takes the request but then does not process it completely, until it times out:

2024-12-02T00:17:16.781179+00:00 kani1 sudo:     cert : PWD=/home/cert ; USER=root ; COMMAND=/usr/bin/systemctl try-reload-or-restart kanidmd
2024-12-02T00:17:16.782933+00:00 kani1 sudo: pam_unix(sudo:session): session opened for user root by (uid=1000)
2024-12-02T00:17:16.788565+00:00 kani1 systemd[1]: Reloading Kanidm Identity Server...
2024-12-02T00:17:16.789862+00:00 kani1 kanidmd[1223]: 00000000-0000-0000-0000-000000000000 INFO     i [info]: Reloaded http tls acceptor
2024-12-02T00:17:16.789891+00:00 kani1 kanidmd[1223]: 00000000-0000-0000-0000-000000000000 INFO     i [info]: Reloaded ldap tls acceptor
...
2024-12-02T00:18:46.889486+00:00 kani1 systemd[1]: kanidmd.service: Reload operation timed out. Killing reload process.
2024-12-02T00:18:46.889587+00:00 kani1 systemd[1]: Reload failed for Kanidm Identity Server.
Actions #1

Updated by crameleon 25 days ago

  • Private changed from Yes to No
Actions #2

Updated by crameleon 25 days ago

It seems reload was only introduced relatively recently: https://github.com/kanidm/kanidm/pull/3144. That'd explain why the issue was not present earlier, as our try-reload-or-restart falls back to restart if a unit does not support reload.

Actions #3

Updated by crameleon 25 days ago

  • Assignee set to firstyear
Actions #4

Updated by firstyear 25 days ago

Actions #5

Updated by firstyear 25 days ago

maybe I'll have to add some sleep statements. Could be reloading too fast for systemd to cope.

Actions #6

Updated by crameleon 25 days ago

I don't think systemd cares how fast it happens, it only wants to receive the notification about the reload being done (READY=1). I do note you have this in the code, but is it ever reaching there?

Actions #7

Updated by firstyear 24 days ago

Yes it does, separate testing shows it working correctly. I suspect race condition.

Actions #8

Updated by firstyear 23 days ago

1.4.4 has a delay here to help systemd, and has extra logging to be extremely clear that we got to the reload point.

Actions

Also available in: Atom PDF