openSUSE admin

I think this and the sudo ticket are actually linked - this error is because the challenge-response session setup is failing as the connection moved from one node to the other. This indicates that:

• The instances are restarting frequently which breaks the chal-rep session establishment
• That the load balancer is not correctly performing sticky sessions
• That something in the load balancer is altering the content of the cookie-jwt which triggers signature failure.

Have there been any changes to the load balancer configuration recently? That would be the first place to look here.

It could be that we're missing the directives below in haproxy (specifically hash-type).

balance source
hash-type consistent

Alternately, we can define a primary/secondary relationship between the pair.

Thanks for the input.

No, there weren't any changes on hel{1,2}, neither in configuration

https://progress.opensuse.org/projects/opensuse-admin/repository/salt/revisions/production/show/pillar/cluster/hel

nor in software since June

crameleon@hel1:/home/crameleon> sudo grep haproxy /var/log/zypp/history|tail -n1
2024-06-21 21:06:51|install|haproxy|2.8.6+git0.f6bd011dc-150600.1.5|x86_64||repo-oss|f6a41993804ab436ab0657eb636158b743d208b18ed8afa3d3e2ee92dea8bacffe3f678d7fd0a45d66200e82a8ea757cef15bd8f099afce8f03a734714d753f0|

balance is already configured correctly: https://progress.opensuse.org/projects/opensuse-admin/repository/salt/revisions/production/entry/pillar/cluster/hel/init.sls#L33.
We can add hash-type.

I think it would be preferable to keep/repair the active/active setup given it worked in the past as well?

https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/2184

Yeah, from my reading balance source isn't deterministic without the hash-type being added.

Committed as
https://progress.opensuse.org/projects/opensuse-admin/repository/salt/revisions/fbdec41c2f77ef93a45d5aa7c4c3e294606d167a
plus minor correction in
https://progress.opensuse.org/projects/opensuse-admin/repository/salt/revisions/853e607039140c29ca78639d6d63be178c534728

Trying to log in the first time after applying the change, I get the same error.

I'm going to need to add some more debugging here, because this still points to a load balancer issue - not a Kanidm issue. Do you have multiple ipv6 private ip's for example?

Can we configure haproxy with x-forward-for?

EDIT: I'm an idiot, it's not ip. It's oauth2. I'm going to follow that path now.

Can you clear your page cookies and retry?

I think I've found the issue now. Sorry for sending you down the complete wrong path.

In the mean time, clear your page cookies and re-try.

For me it's working now, but also I did not encounter it so often before.
Do you still have it happening, @hennevogel ?

Sounds like this is resolved?

IIRC it happened once to me and to @phodac in the last 1.5 weeks.

The kani servercs haven't been updated to 1.4.4 is why :)

I looked yesterday and they are still on 1.4.1