QA » openQA Project » openQA Infrastructure

I guess a retry in a systemd service override would suffice

I see the same error on qam2, so it seems something is broken on the CA side:

Feb 07 01:52:32 qam2 systemd[1]: Starting Certificate Update Runner for Dehydrated...
Feb 07 01:52:32 qam2 dehydrated[25822]: # INFO: Using main config file /etc/dehydrated/config
Feb 07 01:52:32 qam2 dehydrated[25822]: # INFO: Running /usr/bin/dehydrated as dehydrated/dehydrated
Feb 07 01:52:32 qam2 sudo[25822]:     root : PWD=/ ; USER=dehydrated ; GROUP=dehydrated ; COMMAND=/usr/bin/dehydrated --cron
Feb 07 01:52:35 qam2 dehydrated[26142]: HTTP/2 500
Feb 07 01:52:35 qam2 dehydrated[26142]: cache-control: no-store
Feb 07 01:52:35 qam2 dehydrated[26142]: content-type: application/json
Feb 07 01:52:35 qam2 dehydrated[26142]: link: <https://ca-internal.suse.de/acme/acme/directory>;rel="index"
Feb 07 01:52:35 qam2 dehydrated[26142]: replay-nonce: RDc3am9aSmZoMDBpaHdPamo3WVp4U0RweXpnU3F3UU4
Feb 07 01:52:35 qam2 dehydrated[26142]: content-length: 3
Feb 07 01:52:35 qam2 dehydrated[26142]: date: Mon, 07 Feb 2022 00:52:35 GMT
Feb 07 01:52:35 qam2 dehydrated[26142]: 
Feb 07 01:52:35 qam2 dehydrated[26143]: {}
Feb 07 01:52:35 qam2 systemd[1]: dehydrated.service: Main process exited, code=exited, status=1/FAILURE
Feb 07 01:52:35 qam2 systemd[1]: dehydrated.service: Failed with result 'exit-code'.
Feb 07 01:52:35 qam2 systemd[1]: Failed to start Certificate Update Runner for Dehydrated.

The service itself is started from a timer (at least on qam2) with OnCalendar=daily, so there is no need to add retry to the service (which will fail again).

Based on infra investigation and my assumptions, a package upgrade on the STEP-CA side did something to its internal account database, which can be repaired by "recreating the account", I found out it is sufficient to invoke dehydrated --account. After that the renewal works. Apart from that, they have no idea what went wrong (the server side logs are apparently limited).

On the qam2, I did

dehydrated --account
systemctl start dehydrated.service

and it solved the problem.

I moved the old account under /etc/dehydrated/accounts to a backup directory and created a new one via martchus@openqa:~> sudo -u dehydrated dehydrated --register --accept-terms. It now works again and the certificate has been renewed, indeed. (Without "disabling" the old account dehydrated would not create a new one. The new account apparently also has the same hash/checksum/id as the old account.)

so do you have some idea how to recreate the account automatically in case something similar happens again? Just because we don't have dehydrated for that long and this problem already showed up I suggest to look into this.

The alert is off again so. We could try to automate the steps I did in case we get a 500 error but I'm not sure whether it makes generally sense and how to implement it. We'd needed to read the HTTP status code in some "post fail" script.

do we need a wrapper shell script for dehydrated with something like

dehydrated ...
for i in {1..7}; do
    curl -sS https://openqa.suse.de && exit 0
    sudo rm -rf /etc/dehydrated/accounts
    sudo -u dehydrated dehydrated --register --accept-terms
    sudo systemctl restart dehydrated
done

Not sure whether we should forcefully delete all accounts every time we call dehydrated. And restarting the service from itself also doesn't seem to be the best idea. We could add a Restart=… on systemd-level and delete all accounts via ExecStopPost=… if $SERVICE_RESULT is failure.

mkittler wrote:

Not sure whether we should forcefully delete all accounts every time we call dehydrated.

I did not suggest that. I would do that only after the certificate renew fails.

And restarting the service from itself also doesn't seem to be the best idea. We could add a Restart=… on systemd-level and delete all accounts via ExecStopPost=… if $SERVICE_RESULT is failure.

Well, that could an option but we should favor relying less on systemd and instead have something that we could also call, you know, from a container or cloud instance, bootstrap script, you know.

ok

I've also just did the same as on OSD on the monitoring host which had the same problem.

If I understood correctly, our certificates are renewed right now. So I propose to wait for the next renewal round in two weeks and see.

I am not a salt expert so I might ask a wrong question: is it possible to have a different salt task (probably a "state" in salt terminology) which will delete the dehydrated data and then the standard default state which will again install it and obtain the certificates?