action #133583
Updated by mkittler 10 months ago
## Observation
https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1724368 says
```
2023-07-31 10:36:47 INFO * SUSE:Maintenance:30033:304153
2023-07-31 10:36:47 INFO Accepting review for SUSE:Maintenance:29819:304034
2023-07-31 10:36:47 INFO Accepting review for SUSE:Maintenance:29993:304113
2023-07-31 10:36:47 INFO Received 'Not Found'. Request 304113 removed or problem on OBS side, ignoring
2023-07-31 10:36:47 INFO Accepting review for SUSE:Maintenance:29994:304114
…
2023-07-31 10:36:48 INFO Received 'Not Found'. Request 304153 removed or problem on OBS side, ignoring
2023-07-31 10:36:48 INFO End of bot run
++ let 'sleep=BACKOFF_FACTOR*2**count'
++ let count+=1
++ (( count > MAX_RETRIES ))
++ exit 100
Uploading artifacts for failed job 00:01
Uploading artifacts...
bot_*.log: found 3 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created id=1724368 responseStatus=201 Created token=64_LuS46
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 100
```
but I could not identify the underlying cause
## Acceptance criteria
* **AC1**: Those CI jobs no longer run into the issue mentioned under observation or at least retry a reasonably amount of times or ignore the error for good
* **AC2**: The logs make it clear whether an error is fatal or has been ignored or when retries happened
## Suggestions
* It was suggested to log the request body for the 404 because it can have multiple reasons
* Crosscheck the code what should happen before/after "End of bot run" and why the retry is actually triggered
* Improve the error message. It says "ignoring" but it fails
* Check what MAX_RETRIES is set to. Is it 3? Do we want more retries? -> in .gitlab-ci.yml it looks like it is set to 0 and the "exit 100" simply means that the retries are exhausted
* https://github.com/openSUSE/qem-bot/blob/cbef942434e03d1aa92776d27813488a0462f5c1/openqabot/approver.py#L86
* https://github.com/openSUSE/qem-bot/blob/cbef942434e03d1aa92776d27813488a0462f5c1/openqabot/approver.py#L42
* https://gitlab.suse.de/qa-maintenance/bot-ng/-/blob/master/.gitlab-ci.yml#L48