action #179149
openCron <root@ariel> /usr/bin/dehydrated --cron | /opt/os-autoinst-scripts/filter-dehydrated-cron-output "ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/… (Status 503)" size:S
0%
Description
Observation¶
From email
Date: Tue, 18 Mar 2025 00:00:10 +0000 (UTC)
+ ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/chall/1199940347/491383579136/ybvnbQ (Status 503)
Details:
HTTP/2 503
server: nginx
date: Tue, 18 Mar 2025 00:00:10 GMT
content-type: application/problem+json
content-length: 90
cache-control: private
retry-after: 7
{"type": "urn:ietf:params:acme:error:rateLimited", "detail": "Service busy; retry later."}
EXPECTED value GOT EOF
Processing openqa.opensuse.org
+ Checking domain name(s) of existing cert... unchanged.
+ Checking expire date of existing cert...
+ Valid till Apr 16 23:09:37 2025 GMT (Less than 30 days). Renewing!
+ Signing domains...
+ Generating private key...
+ Generating signing request...
+ Requesting new certificate order from CA...
+ Received 1 authorizations URLs from the CA
+ Handling authorization for openqa.opensuse.org
+ 1 pending challenge(s)
+ Deploying challenge tokens...
+ Responding to challenge for openqa.opensuse.org authorization...
Certificate data: o3 says:
Issued On Tuesday, March 18, 2025 at 12:51:08 AM (CET?)
Expires On Monday, June 16, 2025 at 1:51:07 AM
Suggestions¶
- This looks to be a cron job. Can we do systemd service+timer along with retries instead?
- Also look into "filter-dehydrated-cron-output" which we can extend: https://github.com/os-autoinst/scripts/blob/master/filter-dehydrated-cron-output
Updated by okurz 11 days ago
- Subject changed from Cron <root@ariel> /usr/bin/dehydrated --cron | /opt/os-autoinst-scripts/filter-dehydrated-cron-output "ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/… (Status 503)" to Cron <root@ariel> /usr/bin/dehydrated --cron | /opt/os-autoinst-scripts/filter-dehydrated-cron-output "ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/… (Status 503)" size:S
- Description updated (diff)
- Status changed from New to Workable
Updated by ybonatakis 4 days ago
- Status changed from Workable to In Progress
Updated by openqa_review 4 days ago
- Due date set to 2025-04-10
Setting due date based on mean cycle time of SUSE QE Tools
Updated by tinita 4 days ago
Certificate data: o3 says: Issued On Tuesday, March 18, 2025 at 12:51:08 AM (CET?) Expires On Monday, June 16, 2025 at 1:51:07 AM
The cronjob ran at 18 Mar 2025 00:00:10 GMT
Looking into the crontab, the cronjob will run every day.
So this run must have succeeded, so maybe it is doing a retry itself already?
Updated by ybonatakis 4 days ago
I see retry-after: 7
in the request header. I will look at the code again. i thought I didnt see any restart in a request function of dehydrated
Updated by tinita 4 days ago
Also:
/etc/dehydrated/certs/openqa.opensuse.org
-rw------- 1 dehydrated dehydrated 288 Jan 17 00:00 privkey-1737072003.pem
-rw------- 1 dehydrated dehydrated 288 Jan 17 00:08 privkey-1737072484.pem
-rw------- 1 dehydrated dehydrated 288 Mar 18 00:00 privkey-1742256003.pem
-rw------- 1 dehydrated dehydrated 288 Mar 18 00:49 privkey-1742258973.pem
lrwxrwxrwx 1 dehydrated dehydrated 22 Mar 18 00:49 privkey.pem -> privkey-1742258973.pem
So it definitely succeeded on the same day, but the second privkey file suggests that there was some kind of retry
Updated by tinita 4 days ago
- Related to action #165027: (Cron Daemon) sending email about /usr/bin/dehydrated --cron added
Updated by tinita 4 days ago
- Related to action #169078: dashboard.qam.suse.de SSL certificate not deployed within expiry size:S added
Updated by ybonatakis 4 days ago
there are services in both o3 and osd.
the services files are different tho.
So i will close the PR as it doesnt seem to be of any use.
And I will try to find info about those services on the servers.
Updated by ybonatakis 4 days ago
tinita wrote in #note-11:
There is an active dehydrated.service on o3. Just remove the crontab entry.
yes, I wanted to ask this. but also what should I do with the configuration of the existing services. open question is https://github.com/os-autoinst/scripts/pull/385#discussion_r2016297436
Updated by ybonatakis 4 days ago
ok I deleted the crontab row in O3. and seems like someone else did the same on OSD
Updated by ybonatakis 4 days ago
grepping salt-states-openqa there are some results coming up about dehydrated. checking there now
Updated by ybonatakis 3 days ago
- Status changed from In Progress to Feedback
So salt-states-openqa has nothing to do with O3.
O3 and OSD have the dehydrated service active (via timer)
I asked if I can remove the filter-dehydrated-cron-output
completely on the slack and waiting for a confirmation or rejection.
Updated by ybonatakis 3 days ago
I suggested https://github.com/os-autoinst/scripts/pull/386 after the following considerations and actions (some explanation exists in the PR too).
But overall the filtering seems to not be needed as there are ways to get those, with the most direct from the dehydrated itself.
What i have done to keep notification on dehdrated errors is to modify the hook.sh.
Specifically I have editted the /etc/dehydrated/hook.sh, with the following changes
invalid_challenge() -> uncomment printf "Subject: Validation of ${DOMAIN} failed!\n\nOh noez!" | sendmail root
request_failure() -> uncomment printf "Subject: HTTP request failed failed!\n\nA http request failed with status ${STATUSCODE}!" | sendmail root
``if -n $ERROR ; then printf "Subject: dehydrated exits with error: ${ERROR}!" | sendmail root fi
exit_hook() ->
I have tested it injecting _exiterr in some line and I got mail from exit_hook which looks like
From dehydrated@ariel.dmz-prg2.suse.org Fri Mar 28 09:07:21 2025
Return-Path: <dehydrated@ariel.dmz-prg2.suse.org>
X-Original-To: ybonatakis
Delivered-To: ybonatakis@ariel.dmz-prg2.suse.org
Received: by ariel.dmz-prg2.suse.org (Postfix, from userid 473)
id 1FCC319295; Fri, 28 Mar 2025 09:07:21 +0000 (UTC)
Subject: failed with error unchanged_cert hook returned with non-zero exit code!
Message-Id: <20250328090721.1FCC319295@ariel.dmz-prg2.suse.org>
Date: Fri, 28 Mar 2025 09:07:21 +0000 (UTC)
From: dehydrated <dehydrated@ariel.dmz-prg2.suse.org>
I dont really know for sure if we need the entry in exit_hook
the question is whether we want to do this also in OSD? (which I havent checked just yet).
Also I dont know what happens if the system updates.
Updated by ybonatakis 3 days ago
I just want to let a reference to the link Oli shared related to osd https://gitlab.suse.de/search?search=dehydrated&nav_source=navbar&project_id=743&group_id=39&search_code=true&repository_ref=master for future ref