Project

General

Profile

Actions

action #134522

closed

[alert] Certificate renewal on monitor.qa.suse.de might not be working causing alerts size:M

Added by mkittler 10 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
Start date:
2023-08-23
Due date:
2023-09-07
% Done:

0%

Estimated time:
Tags:

Description

Observation

According to https://stats.openqa-monitor.qa.suse.de/d/E9tyiQ17k/ssl-certificate-alerts?orgId=1 the SAN validity of the certificate on monitor.qa.suse.de is going to expire in less than 5 days. I've checked the certificate via martchus@monitor:~> sudo openssl x509 -noout -text -in /etc/dehydrated/certs/monitor.qa.suse.de/fullchain.pem and it is indeed going to expire on Aug 28 22:23:02 2023 GMT. Firefox tells me the same if I access https://stats.openqa-monitor.qa.suse.de.

The dehydrated timer was executed successfuly but a manual run of dehydrated --cron shows that dehydrated just hangs.

Acceptance Criteria

  • AC1: A new certificate is known to be avaialble and used
  • AC2: It is understood what the cause was and how to avoid it next time

Suggestions

  • Check why the certificate expiration is not renewed to be higher than 5 days, e.g. problems with dehydrated.service.
  • Understand why dehydrated hangs and does not produce a new certificate
  • Check if we can make the issue (whatever it is) better visible or workaround automatically in the future

Related issues 1 (1 open0 closed)

Related to openQA Infrastructure - action #134819: Errors in salt minion and master log on osdNew2023-08-30

Actions
Actions #1

Updated by nicksinger 10 months ago

  • Assignee changed from mkittler to nicksinger
  • Priority changed from Normal to Urgent

Given that the certificate will run out in just a few days I bump the prio

Actions #2

Updated by tinita 10 months ago

  • Target version set to Ready
Actions #3

Updated by livdywan 10 months ago

  • Tags changed from alert to alert, infra
  • Subject changed from [alert] Certificate renewal on monitor.qa.suse.de might not be working causing alerts to [alert] Certificate renewal on monitor.qa.suse.de might not be working causing alerts size:M
  • Description updated (diff)
Actions #4

Updated by openqa_review 10 months ago

  • Due date set to 2023-09-07

Setting due date based on mean cycle time of SUSE QE Tools

Actions #5

Updated by nicksinger 10 months ago

  • Status changed from In Progress to Feedback
Actions #6

Updated by nicksinger 10 months ago

  • Assignee deleted (nicksinger)
Actions #7

Updated by nicksinger 10 months ago

  • Assignee set to nicksinger
Actions #8

Updated by livdywan 10 months ago

  • Related to action #134819: Errors in salt minion and master log on osd added
Actions #9

Updated by livdywan 10 months ago

  • Status changed from Feedback to Resolved

High state looks good and certificate renewal seems to be working again as discussed in the unblock

Actions

Also available in: Atom PDF