QA » openQA Project

Observation¶

The pipeline failed yesterday: https://gitlab.suse.de/openqa/openqa-review/-/jobs/552027

Traceback (most recent call last):
1613  File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 438, in _error_catcher
1614    yield
1615  File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 519, in read
1616    data = self._fp.read(amt) if not fp_closed else b""
1617  File "/usr/lib64/python3.8/http/client.py", line 459, in read
1618    n = self.readinto(b)
1619  File "/usr/lib64/python3.8/http/client.py", line 503, in readinto
1620    n = self.fp.readinto(b)
1621  File "/usr/lib64/python3.8/socket.py", line 669, in readinto
1622    return self._sock.recv_into(b)
1623  File "/usr/lib64/python3.8/ssl.py", line 1241, in recv_into
1624    return self.read(nbytes, buffer)
1625  File "/usr/lib64/python3.8/ssl.py", line 1099, in read
1626    return self._sslobj.read(len, buffer)
1627ConnectionResetError: [Errno 104] Connection reset by peer
1628During handling of the above exception, another exception occurred:
1629Traceback (most recent call last):
1630  File "/usr/lib/python3.8/site-packages/requests/models.py", line 753, in generate
1631    for chunk in self.raw.stream(chunk_size, decode_content=True):
1632  File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 576, in stream
1633    data = self.read(amt=amt, decode_content=decode_content)
1634  File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 541, in read
1635    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
1636  File "/usr/lib64/python3.8/contextlib.py", line 131, in __exit__
1637    self.gen.throw(type, value, traceback)
1638  File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 455, in _error_catcher
1639    raise ProtocolError("Connection broken: %r" % e, e)
1640urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
1641During handling of the above exception, another exception occurred:
1642Traceback (most recent call last):
1643  File "/usr/bin/openqa-review", line 33, in <module>
1644    sys.exit(load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')())
1645  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1557, in main
1646    report = generate_report(args)
1647  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1488, in generate_report
1648    return Report(browser, args, root_url, job_groups)
1649  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1437, in __init__
1650    self.report[k] = self._one_report(v)
1651  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1448, in _one_report
1652    return ProductReport(self.browser, job_group_url, self.root_url, self.args)
1653  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1162, in __init__
1654    self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser)
1655  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 898, in __init__
1656    self._search_for_bugrefs_for_softfailures(results)
1657  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 942, in _search_for_bugrefs_for_softfailures
1658    module_url = self._get_url_to_softfailed_module(v["href"])
1659  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 984, in _get_url_to_softfailed_module
1660    details = self.test_browser.get_json("/api/v1/jobs/" + job_url.split("/")[-1] + "/details")
1661  File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 77, in get_json
1662    return self.get_page(url, as_json=True, cache=cache)
1663  File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 104, in get_page
1664    content = self._get(absolute_url, as_json=as_json)
1665  File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 119, in _get
1666    r = http.get(url, auth=self.auth, timeout=30, headers=self.headers)
1667  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
1668    return self.request('GET', url, **kwargs)
1669  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
1670    resp = self.send(prep, **send_kwargs)
1671  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 697, in send
1672    r.content
1673  File "/usr/lib/python3.8/site-packages/requests/models.py", line 831, in content
1674    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
1675  File "/usr/lib/python3.8/site-packages/requests/models.py", line 756, in generate
1676    raise ChunkedEncodingError(e)
1677requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

I guess only a retry would help here, but also the error message can possible be made nicer.

Acceptance criteria¶

• AC1: At least 10 jobs in the openqa-review pipeline pass without this problem after a potential fix is applied

Suggestions¶

• Figure out what "104" in relation to the message "Connection reset by peer" could mean
• Try to apply "retry" or actual "reconnect" at the right step as we know that this is a sporadic issue so workaround like this should work

Workaround¶

Retrigger