Project

General

Profile

action #69613

osd-pre-deployment checks fail due to invalid certificates for stats.openqa-monitor.qa.suse.de

Added by okurz 12 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2020-08-05
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://gitlab.suse.de/openqa/osd-deployment/-/jobs/240416 shows

$ eval "$GRAFANA_ALERTS" > current_alerts
ERROR: Job failed: command terminated with exit code 1

meaning that the curl-command itself failed.

History

#1 Updated by okurz 12 months ago

  • Status changed from New to Feedback
  • Assignee set to okurz
  • Priority changed from Normal to Urgent
  • Target version set to Ready

https://infra.nue.suse.com/SelfService/Display.html?id=175585 is a related ticket by nicksinger asking for updated certificates and now stats.openqa-monitor.qa.suse.de is not properly reachable anymore, coincidence? Asked jdsn in chat https://chat.suse.de/channel/suse-it-ama?msg=aHmkC28BAoMR2J824

#2 Updated by okurz 12 months ago

  • Priority changed from Urgent to High

No response in RC nor in https://infra.nue.suse.com/SelfService/Display.html?id=175585 :(

It looks like the current problem with certificate was triggered by nsinger who did changes on the openqa-monitor host as described in #68785#note-8 but I could not find the information within the host itself directly, e.g. no entry in bash history of "root" nor "nsinger". Only that the command last tells me that he did login 2020-08-04 and we see changes:

root@openqa-monitor:/etc/ssl # ls -ltra
total 36
lrwxrwxrwx   1 root root    28 Aug  1  2019 certs -> /var/lib/ca-certificates/pem
lrwxrwxrwx   1 root root    38 Aug  1  2019 ca-bundle.pem -> /var/lib/ca-certificates/ca-bundle.pem
drwx------   2 root root  4096 Jan 10  2020 private
-rw-r--r--   1 root root 10771 Jan 10  2020 openssl.cnf
drwxr-xr-x   4 root root  4096 Jan 10  2020 .
drwxr-xr-x 105 root root 12288 Aug  4 03:02 ..
drwxr-x---   2 root root  4096 Aug  4 19:53 stats.openqa-monitor.qa.suse.de
root@openqa-monitor:/etc/ssl # ls -ltra *
lrwxrwxrwx 1 root root    28 Aug  1  2019 certs -> /var/lib/ca-certificates/pem
lrwxrwxrwx 1 root root    38 Aug  1  2019 ca-bundle.pem -> /var/lib/ca-certificates/ca-bundle.pem
-rw-r--r-- 1 root root 10771 Jan 10  2020 openssl.cnf

private:
total 8
drwx------ 2 root root 4096 Jan 10  2020 .
drwxr-xr-x 4 root root 4096 Jan 10  2020 ..

stats.openqa-monitor.qa.suse.de:
total 48
-rw------- 1 root root 3276 Aug 28  2019 cert.key
-rw-r----- 1 root root 2569 Sep 10  2019 cert.crt
-rw-r--r-- 1 root root 4935 Sep 10  2019 suse_chain.crt
-rw-r--r-- 1 root root 7504 Sep 10  2019 fullchain.crt
drwxr-xr-x 4 root root 4096 Jan 10  2020 ..
-rw-r--r-- 1 root root  422 Aug  4 09:56 req.conf
-rw-r--r-- 1 root root 1890 Aug  4 09:56 cert.csr
drwxr-x--- 2 root root 4096 Aug  4 19:53 .
-rw-r--r-- 1 root root 7500 Aug  4 19:53 monitor.qa.suse.de.chained.crt

just additionally putting the files from https://infra.nue.suse.com/SelfService/Display.html?id=175585 does not seem to help.

maybe I can just revert changes in the directory /etc/ssl/stats.openqa-monitor.qa.suse.de but I do not know how. Asked in ticket for help additionally again.

Created workaround for deployment with https://gitlab.suse.de/openqa/osd-deployment/-/merge_requests/21 now and triggered https://gitlab.suse.de/openqa/osd-deployment/-/pipelines/73517

#3 Updated by okurz 12 months ago

  • Status changed from Feedback to Blocked

#4 Updated by okurz 11 months ago

  • Status changed from Blocked to Resolved

https://infra.nue.suse.com/SelfService/Display.html?id=175585 was resolved. We have received a new crt file. I copied the file over to openqa-monitor.qa:/etc/ssl/stats.openqa-monitor.qa.suse.de and on openqa-monitor.qa in /etc/ssl/stats.openqa-monitor.qa.suse.de I did:

cat monitor.qa.suse.de.crt SUSE_CA_suse.de.crt SUSE_CA_Root.crt > cert.crt

chromium says that https://stats.openqa-monitor.qa.suse.de/ is ok now. Also for verification nsinger suggested echo | openssl s_client -showcerts -connect 10.162.0.21:443 -servername stats.openqa-monitor.qa.suse.de which looks fine as well. Also fine: https://monitor.qa.suse.de/, https://stats.monitor.qa.suse.de/, https://openqa-monitor.qa.suse.de

Also available in: Atom PDF