Project

General

Profile

Actions
Copy link

action #168091

closed

logwarn: "Publishing opensuse.openqa.job.done failed: SSL connect attempt failed"

Added by nicksinger 7 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2024-10-10
Due date:
% Done:

0%

Estimated time:
Tags:
o3, reactive work, ssl

Description

Observation

We received an e-mail from logwarn about the following errors in /var/log/openqa:

[2024-10-10T08:34:46.593842Z] [error] [pid:4744] Publishing opensuse.openqa.job.done failed: SSL connect attempt failed error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
[2024-10-10T08:34:47.611514Z] [error] [pid:4744] Publishing opensuse.openqa.job.done failed: SSL connect attempt failed error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
…
[2024-10-10T08:35:45.683230Z] [error] [pid:26453] Publishing opensuse.openqa.job.done failed: SSL connect attempt failed error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
[2024-10-10T08:35:46.012665Z] [error] [pid:24209] Publishing opensuse.openqa.job.done failed: SSL connect attempt failed error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

the direct issue seems to be gone but I can't even verify or understand what is wrong because the error message lacks details. So at the very least it should be improved (e.g. which host it tried to connect to? What FQDN?)

Suggestions

  • The opensuse part suggests this is rabbit.opensuse.org
  • Disambiguate the log message to clarify if this is an issue on e.g. the openQA side or somewhere else?
    • See lib/OpenQA/WebAPI/Plugin/AMQP.pm
  • Add the message to the blocklist? - Better not as these error message could in general be actionable for us.

Related issues 2 (0 open — 2 closed)

Related to openQA Infrastructure (public) - action #169690: logwarn repeatedly sending openssl related problems. Is the state file stuck?Resolvedokurz2024-11-11

Actions
Copied to openQA Project (public) - action #178147: ariel.suse-dmz.opensuse.org: " Publishing opensuse.openqa.comment.create failed: Can't connect: System error (9 attempts left)" size:SResolvedmkittler

Actions
Actions
Copy link
 #1

Updated by ybonatakis 7 months ago

  • Tags set to infra
  • Category set to Regressions/Crashes
  • Target version set to Tools - Next
Actions
Copy link
 #2

Updated by jbaier_cz 7 months ago

"Publishing opensuse.openqa.job.done failed" looks like amqp messages, i.e. that would be connection to rabbit.opensuse.org.

Actions
Copy link
 #3

Updated by okurz 7 months ago

  • Tags changed from infra to o3, ssl
  • Target version changed from Tools - Next to Ready
Actions
Copy link
 #4

Updated by mkittler 7 months ago

Yes, for o3 it would be rabbit.opensuse.org and for OSD rabbit.suse.de. It is basically what's in the url field of the [amqp] section of openqa.ini.

Actions
Copy link
 #5

Updated by okurz 6 months ago

  • Description updated (diff)
  • Status changed from New to Rejected
  • Assignee set to okurz

in theory we could add the FQDN in the log message but during the estimation call we decided that it's not worth the effort for such problem which is not likely to reappear anytime soon.

Actions
Copy link
 #6

Updated by ybonatakis 5 months ago

  • Status changed from Rejected to Workable

I reopened it as there are a batch of new emails for that issue

Actions
Copy link
 #7

Updated by okurz 5 months ago

  • Status changed from Workable to New
  • Assignee deleted (okurz)
Actions
Copy link
 #8

Updated by jbaier_cz 5 months ago

In the early morning openssl s_client -connect rabbit.opensuse.org:amqps showed expired certificate; it looks good now and actually the presented certificate is not entirely new, maybe there was a missed reload after renewal like we saw in #169078

Actions
Copy link
 #9

Updated by okurz 5 months ago · Edited

  • Status changed from New to Resolved
  • Assignee set to okurz

Right. Thanks for looking into this. No more problem observed. Although no clear solution was stated by according administrators. I also asked in https://app.element.io/?updated=1.11.69#/room/#admin:opensuse.org within message https://matrix.to/#/!wZwXBKKJqHykbBnWhM:opensuse.org/$Lstp16Fy6kJnF75aPPqkrRB2f9DBdy6iYnmkLT9Pvi0 and got a response pointing to #168487

Actions
Copy link
 #10

Updated by tinita 5 months ago

  • Related to action #169690: logwarn repeatedly sending openssl related problems. Is the state file stuck? added
Actions
Copy link
 #11

Updated by gpathak about 2 months ago

  • Copied to action #178147: ariel.suse-dmz.opensuse.org: " Publishing opensuse.openqa.comment.create failed: Can't connect: System error (9 attempts left)" size:S added
Actions
Copy link

Also available in: Atom PDF