Project

General

Profile

Actions

action #152741

closed

[tools] gitlab CI - openqa_review failed with connection timeout on osd

Added by osukup 4 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2023-12-18
Due date:
% Done:

0%

Estimated time:

Description

Looks like osd wasn't able to reply to api longer than 30 sec, can be a random network problem or too complicated query?

https://gitlab.suse.de/openqa/openqa-review/-/jobs/2077520

usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old|EOL)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.1LmmaKoNx7 --job-groups '^SLE.*15.*(Functional)'
..............................WARNING:urllib3.connectionpool:Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=30)")': /api/v1/parent_groups
..................................WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=30)")': /api/v1/parent_groups
......................................WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=30)")': /api/v1/parent_groups
............................................................WARNING:urllib3.connectionpool:Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=30)")': /api/v1/parent_groups
..............................................................WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=30)")': /api/v1/parent_groups
..............................................................................................WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=30)")': /api/v1/parent_groups
...................................................................................................................................................................WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=30)")': /api/v1/parent_groups
..............................WARNING:openqa_review.browser:Request to https://openqa.suse.de/api/v1/parent_groups was not successful after 7 retries: HTTPSConnectionPool(host='openqa.suse.de', port=443): Max retries exceeded with url: /api/v1/parent_groups (Caused by ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=30)"))
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 782, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 470, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 514, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/lib64/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
TimeoutError: _ssl.c:989: The handshake operation timed out

The above exception was the direct cause of the following exception:
Actions #1

Updated by okurz 4 months ago

  • Tags set to reactive work
  • Priority changed from Normal to High
  • Target version set to Ready

Do you have a URL to the failed CI Job please?

Actions #2

Updated by livdywan 4 months ago

  • Description updated (diff)
Actions #3

Updated by livdywan 4 months ago

  • Status changed from New to In Progress
  • Assignee set to livdywan
Actions #4

Updated by livdywan 4 months ago

  • Status changed from In Progress to Feedback

livdywan wrote in #note-3:

https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/20

Reviewed and merged. Let's see how this goes

Actions #5

Updated by livdywan 4 months ago

  • Status changed from Feedback to Resolved

livdywan wrote in #note-4:

livdywan wrote in #note-3:

https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/20

Reviewed and merged. Let's see how this goes

Seems like it's fine

Actions

Also available in: Atom PDF