Project

General

Profile

action #97856

Updated by okurz about 3 years ago

## Observation 

 The pipeline failed yesterday: https://gitlab.suse.de/openqa/openqa-review/-/jobs/552027 
 ``` 
 Traceback (most recent call last): 
 1613    File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 438, in _error_catcher 
 1614      yield 
 1615    File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 519, in read 
 1616      data = self._fp.read(amt) if not fp_closed else b"" 
 1617    File "/usr/lib64/python3.8/http/client.py", line 459, in read 
 1618      n = self.readinto(b) 
 1619    File "/usr/lib64/python3.8/http/client.py", line 503, in readinto 
 1620      n = self.fp.readinto(b) 
 1621    File "/usr/lib64/python3.8/socket.py", line 669, in readinto 
 1622      return self._sock.recv_into(b) 
 1623    File "/usr/lib64/python3.8/ssl.py", line 1241, in recv_into 
 1624      return self.read(nbytes, buffer) 
 1625    File "/usr/lib64/python3.8/ssl.py", line 1099, in read 
 1626      return self._sslobj.read(len, buffer) 
 1627ConnectionResetError: [Errno 104] Connection reset by peer 
 1628During handling of the above exception, another exception occurred: 
 1629Traceback (most recent call last): 
 1630    File "/usr/lib/python3.8/site-packages/requests/models.py", line 753, in generate 
 1631      for chunk in self.raw.stream(chunk_size, decode_content=True): 
 1632    File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 576, in stream 
 1633      data = self.read(amt=amt, decode_content=decode_content) 
 1634    File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 541, in read 
 1635      raise IncompleteRead(self._fp_bytes_read, self.length_remaining) 
 1636    File "/usr/lib64/python3.8/contextlib.py", line 131, in __exit__ 
 1637      self.gen.throw(type, value, traceback) 
 1638    File "/usr/lib/python3.8/site-packages/urllib3/response.py", line 455, in _error_catcher 
 1639      raise ProtocolError("Connection broken: %r" % e, e) 
 1640urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')) 
 1641During handling of the above exception, another exception occurred: 
 1642Traceback (most recent call last): 
 1643    File "/usr/bin/openqa-review", line 33, in <module> 
 1644      sys.exit(load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')()) 
 1645    File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1557, in main 
 1646      report = generate_report(args) 
 1647    File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1488, in generate_report 
 1648      return Report(browser, args, root_url, job_groups) 
 1649    File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1437, in __init__ 
 1650      self.report[k] = self._one_report(v) 
 1651    File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1448, in _one_report 
 1652      return ProductReport(self.browser, job_group_url, self.root_url, self.args) 
 1653    File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1162, in __init__ 
 1654      self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser) 
 1655    File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 898, in __init__ 
 1656      self._search_for_bugrefs_for_softfailures(results) 
 1657    File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 942, in _search_for_bugrefs_for_softfailures 
 1658      module_url = self._get_url_to_softfailed_module(v["href"]) 
 1659    File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 984, in _get_url_to_softfailed_module 
 1660      details = self.test_browser.get_json("/api/v1/jobs/" + job_url.split("/")[-1] + "/details") 
 1661    File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 77, in get_json 
 1662      return self.get_page(url, as_json=True, cache=cache) 
 1663    File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 104, in get_page 
 1664      content = self._get(absolute_url, as_json=as_json) 
 1665    File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 119, in _get 
 1666      r = http.get(url, auth=self.auth, timeout=30, headers=self.headers) 
 1667    File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 555, in get 
 1668      return self.request('GET', url, **kwargs) 
 1669    File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 542, in request 
 1670      resp = self.send(prep, **send_kwargs) 
 1671    File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 697, in send 
 1672      r.content 
 1673    File "/usr/lib/python3.8/site-packages/requests/models.py", line 831, in content 
 1674      self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' 
 1675    File "/usr/lib/python3.8/site-packages/requests/models.py", line 756, in generate 
 1676      raise ChunkedEncodingError(e) 
 1677requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')) 
 ``` 

 I guess only a retry would help here, but also the error message can possible be made nicer. 

 ## Acceptance criteria 
 * **AC1:** At least 10 jobs in the openqa-review pipeline pass without this problem after a potential fix is applied 

 ## Suggestions 
 * Figure out what "104" in relation to the message "Connection reset by peer" could mean 
 * Try to apply "retry" or actual "reconnect" at the right step as we know that this is a sporadic issue so workaround like this should work 

 ## Workaround 
 Retrigger

Back