Actions
action #154546
closedCron fetch_openqa_bugs refused or timed out trying to fetch individual tickets
Start date:
2023-10-23
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Observation¶
[...]
socket.timeout: timed out
During handling of the above exception, another exception occurred:
[...]
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7fdf94b4b4a8>, 'Connection to progress.opensuse.org timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
[...]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='progress.opensuse.org', port=443): Max retries exceeded with url: /issues/31576.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fdf94b4b4a8>, 'Connection to progress.opensuse.org timed out. (connect timeout=60)'))
During handling of the above exception, another exception occurred:
[...]
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='progress.opensuse.org', port=443): Max retries exceeded with url: /issues/31576.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fdf94b4b4a8>, 'Connection to progress.opensuse.org timed out. (connect timeout=60)'))
or
[...]
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
[...]
File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f6c6acae588>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
[...]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='progress.opensuse.org', port=443): Max retries exceeded with url: /issues/150923.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6c6acae588>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
[...]
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='progress.opensuse.org', port=443): Max retries exceeded with url: /issues/150923.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6c6acae588>: Failed to establish a new connection: [Errno 111] Connection refused',))
Suggestions¶
- Add (more) retries, or consider exponential backoff
- Ignore errors in the context of bug fetching - they are not typially actionable anyway
- Come up with another way to alert us of on-going lack of updates of the bugs database
Updated by livdywan 3 months ago
- Related to action #152470: openqa-service fetch_openqa_bugs "requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='bugzilla.suse.com', port=443)" added
Updated by okurz 3 months ago
- Due date set to 2024-02-14
- Status changed from In Progress to Feedback
I have conducted the cron scripts on openqa-service
*/10 * * * * root (date;fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log
1 */10 * * * root (date;fetch_openqa_bugs /etc/openqa/bugfetcher_o3.conf) > /tmp/fetch_openqa_bugs_o3.log
and they were quickly executing so I assume that changes to progress.o.o fixed that. Setting ticket back to previous status.
As the problem was clearly on the side of the external service and it was resolved quick enough I would not worry more about it. Regarding "too many emails" we could of course temporarily disable such services until the external problem is resolved.
I will wait if any further problems arise until tomorrow and if not then resolve after that.
Actions