action #95822: qa-maintenance/openQABot failed to trigger aggregate tests with "urllib.error.HTTPError: HTTP Error 500: Internal Server Error" - QA (public) - openSUSE Project Management Tool

action #95822

## Observation 

 From https://chat.suse.de/channel/qem-openqa-review?msg=ifWGbs7QXfJdGJqTf 

 From `ssh qam2 'journalctl -M openqabot -u openqabot-full --since=2021-07-22'`: 

 ``` 
 Jul 22 01:09:50 openqabot oqaqambot[21718]: INFO: Updates shedule enabled for this run on PUBCLOUD12SP5AZUREStandardgen2:x86_64 
 Jul 22 01:09:50 openqabot oqaqambot[21718]: INFO: sle-12-SP5-x86_64 repohash: 4a870c348452ec6fb6c9ca52b30d9aea 
 Jul 22 01:09:50 openqabot oqaqambot[21718]: INFO: Incidents in sle-12-SP5-x86_64: {'ARCH': 'x86_64', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'BUILD': '20210722-1', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'DISTRI': 'sle', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'FLAVOR': 'AZURE-Standard-gen2-Updates', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'OS_TEST_ISSUES': '20175,20204,20222,20248,20258,20283,20344,20353,20354,20431,20434,20450,20475,20477,20485,4705', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'PUBLICCLOUD_TOOLS_IMAGE_QUERY': 'https://openqa.suse.de/group_overview/276.json', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'PUBLIC_CLOUD_AZURE_OFFER': 'sles-12-sp5', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'PUBLIC_CLOUD_AZURE_SKU': 'gen2', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'PUBLIC_CLOUD_IMAGE_ID': '', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'REPOHASH': '4a870c348452ec6fb6c9ca52b30d9aea', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'SDK_TEST_ISSUES': '20204,20222,20225,20248,20274,20283,20326,20344,20354,20434,20450,20475,20477', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    'VERSION': '12-SP5', 
 Jul 22 01:09:50 openqabot oqaqambot[21718]:    '_OBSOLETE': 1} 
 Jul 22 01:09:57 openqabot oqaqambot[21718]: WARNING: PUBCLOUD12SP5AZUREBasic is outdated: 20210721-1 
 Jul 22 01:09:57 openqabot oqaqambot[21718]: INFO: Updates shedule enabled for this run on PUBCLOUD12SP5AZUREBasic:x86_64 
 Jul 22 01:09:57 openqabot oqaqambot[21718]: INFO: sle-12-SP5-x86_64 repohash: 4a870c348452ec6fb6c9ca52b30d9aea 
 Jul 22 01:09:57 openqabot oqaqambot[21718]: INFO: Incidents in sle-12-SP5-x86_64: {'ARCH': 'x86_64', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'BUILD': '20210722-1', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'DISTRI': 'sle', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'FLAVOR': 'AZURE-Basic-Updates', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'OS_TEST_ISSUES': '20175,20204,20222,20248,20258,20283,20344,20353,20354,20431,20434,20450,20475,20477,20485,4705', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'PUBLICCLOUD_TOOLS_IMAGE_QUERY': 'https://openqa.suse.de/group_overview/276.json', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'PUBLIC_CLOUD_AZURE_OFFER': 'sles-12-sp5-basic', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'PUBLIC_CLOUD_AZURE_SKU': 'gen1', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'PUBLIC_CLOUD_IMAGE_ID': '', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'REPOHASH': '4a870c348452ec6fb6c9ca52b30d9aea', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'SDK_TEST_ISSUES': '20204,20222,20225,20248,20274,20283,20326,20344,20354,20434,20450,20475,20477', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    'VERSION': '12-SP5', 
 Jul 22 01:09:57 openqabot oqaqambot[21718]:    '_OBSOLETE': 1} 
 Jul 22 01:10:02 openqabot oqaqambot[21718]: Traceback (most recent call last): 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/bin/oqaqambot", line 11, in <module> 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       load_entry_point('openQABot==0.3.0', 'console_scripts', 'oqaqambot')() 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/openqabot/main.py", line 18, in main 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       sys.exit(run_bot(logger, args, sys)) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/openqabot/main.py", line 41, in run_bot 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       return OpenQABot(metadata, args)() 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/openqabot/openqabot.py", line 110, in __call__ 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       self.calculate_updates() 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/openqabot/openqabot.py", line 142, in calculate_updates 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       incidents = updates.gather_incidents(self.apiurl, arch) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/openqabot/update/updates.py", line 109, in gather_incidents 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       req = self.is_incident_in_testing(apiurl, incident) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/openqabot/update/updates.py", line 59, in is_incident_in_testing 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       res = osc.core.search(apiurl, request=xpath)["request"] 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/osc/core.py", line 6819, in search 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       f = http_GET(u) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/osc/core.py", line 3421, in http_GET 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       def http_GET(*args, **kwargs):      return http_request('GET', *args, **kwargs) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib/python3.6/site-packages/osc/core.py", line 3410, in http_request 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       fd = urlopen(req, data=data) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       return opener.open(url, data, timeout) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib64/python3.6/urllib/request.py", line 532, in open 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       response = meth(req, response) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       'http', request, response, code, msg, hdrs) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib64/python3.6/urllib/request.py", line 570, in error 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       return self._call_chain(*args) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       result = func(*args) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:     File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default 
 Jul 22 01:10:02 openqabot oqaqambot[21718]:       raise HTTPError(req.full_url, code, msg, hdrs, fp) 
 Jul 22 01:10:02 openqabot oqaqambot[21718]: urllib.error.HTTPError: HTTP Error 500: Internal Server Error 
 Jul 22 01:10:03 openqabot systemd[1]: openqabot-full.service: Main process exited, code=exited, status=1/FAILURE 
 Jul 22 01:10:03 openqabot systemd[1]: Failed to start Schedule and review Maintenance incidents in openQA full run. 
 ``` 

 ## Problem 

 Could be a temporary performance problem on openqa.suse.de. In any case a retry should be conducted. 

 ## Suggestions 

 * Logs on openqa.suse.de should be checked for the same timestamp (beware of timezone differences) 
 * Implement retry, potentially even possible on systemd service level but that could cause lots of redundant jobs triggered if there is just a minor failure *after* many jobs had been triggered 

 ## Workaround 
 On qam2: Trigger `systemctl -M openqabot start openqabot-full` manually. Caution: Takes multiple minutes, better do that in a screen session and monitor `journalctl -M openqabot -u openqabot-full -f`

Back

Project

General

Profile

QA (public)

action #95822