Project

General

Profile

Actions
Copy link

action #117526

closed

[alert] Dehydrated fails again on OSD

Added by mkittler over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
mkittler
Category:
-
Target version:
openQA Project - Ready
Start date:
2022-10-04
Due date:
% Done:

0%

Estimated time:
Tags:
alert

Description

Observation

martchus@openqa:~> sudo  journalctl -fu dehydrated.service
Okt 03 01:16:17 openqa systemd[1]: dehydrated.service: Deactivated successfully.
Okt 03 01:16:17 openqa systemd[1]: Finished Certificate Update Runner for Dehydrated.
Okt 04 00:15:14 openqa systemd[1]: Starting Certificate Update Runner for Dehydrated...
Okt 04 00:15:14 openqa dehydrated[9335]: # INFO: Using main config file /etc/dehydrated/config
Okt 04 00:15:14 openqa dehydrated[9335]: # INFO: Using additional config file /etc/dehydrated/config.d/suse-ca.sh
Okt 04 00:15:14 openqa dehydrated[9335]: # INFO: Running /usr/bin/dehydrated as dehydrated/dehydrated
Okt 04 00:15:14 openqa sudo[9335]:     root : PWD=/ ; USER=dehydrated ; GROUP=dehydrated ; COMMAND=/usr/bin/dehydrated --cron
Okt 04 00:15:16 openqa systemd[1]: dehydrated.service: Main process exited, code=exited, status=1/FAILURE
Okt 04 00:15:16 openqa systemd[1]: dehydrated.service: Failed with result 'exit-code'.
Okt 04 00:15:16 openqa systemd[1]: Failed to start Certificate Update Runner for Dehydrated.

Just restarting doesn't help:

martchus@openqa:~> sudo systemctl restart dehydrated.service
Job for dehydrated.service failed because the control process exited with error code.
See "systemctl status dehydrated.service" and "journalctl -xeu dehydrated.service" for details.
martchus@openqa:~> sudo  journalctl -fu dehydrated.service
Okt 04 00:15:16 openqa systemd[1]: Failed to start Certificate Update Runner for Dehydrated.
Okt 04 11:13:20 openqa systemd[1]: Starting Certificate Update Runner for Dehydrated...
Okt 04 11:13:20 openqa dehydrated[8045]: # INFO: Using main config file /etc/dehydrated/config
Okt 04 11:13:20 openqa dehydrated[8045]: # INFO: Using additional config file /etc/dehydrated/config.d/suse-ca.sh
Okt 04 11:13:20 openqa curl[8088]: looking for plugins in '/usr/lib64/sasl2
Okt 04 11:13:20 openqa dehydrated[8045]: # INFO: Running /usr/bin/dehydrated as dehydrated/dehydrated
Okt 04 11:13:20 openqa sudo[8045]:     root : PWD=/ ; USER=dehydrated ; GROUP=dehydrated ; COMMAND=/usr/bin/dehydrated --cron
Okt 04 11:13:20 openqa systemd[1]: dehydrated.service: Main process exited, code=exited, status=1/FAILURE
Okt 04 11:13:20 openqa systemd[1]: dehydrated.service: Failed with result 'exit-code'.
Okt 04 11:13:20 openqa systemd[1]: Failed to start Certificate Update Runner for Dehydrated.

Further information

Actions
Copy link
 #1

Updated by mkittler over 1 year ago

There are also related salt pipeline failures:

          ID: dehydrated --register --accept-terms
    Function: cmd.run
      Result: False
     Comment: Command "dehydrated --register --accept-terms" run
     Started: 11:57:28.849646
    Duration: 1436.846 ms
     Changes:   
              ----------
              pid:
                  2571
              retcode:
                  1
              stderr:
                  ERROR: Problem connecting to server (get for https://ca-internal.suse.de/acme/acme/directory; curl returned with 7)
                  EXPECTED value GOT EOF
              stdout:
                  # INFO: Using main config file /etc/dehydrated/config
                  # INFO: Using additional config file /etc/dehydrated/config.d/suse-ca.sh

(e.g. https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1170464)

Actions
Copy link
 #2

Updated by mkittler over 1 year ago

  • Status changed from New to Feedback
  • Assignee set to mkittler

Looks like https://ca-internal.suse.de/ is completely down. At least I cannot reach it via VPN or from OSD. I'll check/ask on slack.

Actions
Copy link
 #3

Updated by mkittler over 1 year ago

  • Status changed from Feedback to Resolved

It works again after asking in the infra channel and people recovering the service:

martchus@openqa:~> sudo systemctl status dehydrated.service
○ dehydrated.service - Certificate Update Runner for Dehydrated
     Loaded: loaded (/usr/lib/systemd/system/dehydrated.service; static)
     Active: inactive (dead) since Tue 2022-10-04 16:19:45 CEST; 7s ago
TriggeredBy: ● dehydrated.timer
    Process: 11706 ExecStart=/usr/bin/dehydrated --cron (code=exited, status=0/SUCCESS)
    Process: 12014 ExecStartPost=/usr/bin/find -L /etc/dehydrated/postrun-hooks.d -maxdepth 1 -executable -type f -exec {} ; (code=exited, status=0/SUCCESS)
   Main PID: 11706 (code=exited, status=0/SUCCESS)

Okt 04 16:19:43 openqa systemd[1]: Starting Certificate Update Runner for Dehydrated...
Okt 04 16:19:43 openqa dehydrated[11706]: # INFO: Using main config file /etc/dehydrated/config
Okt 04 16:19:43 openqa dehydrated[11706]: # INFO: Using additional config file /etc/dehydrated/config.d/suse-ca.sh
Okt 04 16:19:44 openqa curl[11725]: looking for plugins in '/usr/lib64/sasl2
Okt 04 16:19:44 openqa dehydrated[11706]: # INFO: Running /usr/bin/dehydrated as dehydrated/dehydrated
Okt 04 16:19:44 openqa sudo[11706]:     root : PWD=/ ; USER=dehydrated ; GROUP=dehydrated ; COMMAND=/usr/bin/dehydrated --cron
Okt 04 16:19:45 openqa systemd[1]: dehydrated.service: Deactivated successfully.
Okt 04 16:19:45 openqa systemd[1]: Finished Certificate Update Runner for Dehydrated.

Georg said

online again.. for now

so maybe we'll run into the problem again. However, I suppose there's nothing we can do from our side for now so I'll close the ticket.

Actions
Copy link

Also available in: Atom PDF