action #128129
closedcodecov checks missing in pull requests size:M
0%
Description
Observation¶
It seems multiple, maybe all, pull requests don't have codecov information. https://github.com/os-autoinst/openQA/pull/5090#issuecomment-1517387110 is one example. It seems only openQA fork-based pull requests are affected, not os-autoinst and also not pull requests from within the main repo.
OBS is also affected since some days, see their upstream report on https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266 . Problems likely started for us at the same time so either something on the side of codecov or circleCI or github changed.
Specific details from failing jobs:
https://app.circleci.com/pipelines/github/os-autoinst/openQA/11317/workflows/32a8f778-4ac3-429f-b4bb-6efc15a7cd15/jobs/105794?invite=true#step-112-43
[2023-04-21T09:17:36.470Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.5.0&token=*******&branch=pull%2F5088&build=105794&build_url=https%3A%2F%2Fcircleci.com%2Fgh%2Fos-autoinst%2FopenQA%2F105794&commit=a9a899bbdb18238728d38783e3ed47f2880a55b7&job=0&pr=5088&service=circleci&slug=os-autoinst%2FopenQA&name=&tag=&flags=&parent=
[2023-04-21T09:17:36.839Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Commit sha does not match Circle build. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
CircleCI received exit code 0
We have a circleCI token in the main repo but that is not accessible to forks. For os-autoinst fork-based PRs without access to a token also still seem to work where we use the github action that works.
Steps to reproduce¶
Likely any pull request created will not have codecov check results recorded
Expected result¶
- Pull requests for os-autoinst+openQA should have code coverage information
- PRs like "dependency cron" are merged automatically by mergify again
https://app.circleci.com/pipelines/github/os-autoinst/openQA/11371/workflows/d84e5d02-061e-4679-acdb-e2950b917ec8/jobs/106294 is a working example from a recent origin-based PR after the problem happened.
Suggestions¶
- DONE Research about the problem if others are hit as well
- DONE Try to reproduce the problem
- Different alternatives we can follow:
1. Continue investigation to find a workaround as https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266 is not progressing. Maybe we find other github projects also using circleCI+codecov and they might have found better workarounds
* For this trying with "circleCI orb": https://github.com/os-autoinst/openQA/pull/5110
2. Enable the use of codeclimate additionally to codecov, same as OBS did. Maybe also coveralls
3. Migrate from circleCI to github action – at least to crosscheck
Further details¶
See #127688, maybe related? - most likely not, see #12812#note-1
Out of scope¶
Instead of relying on any external coverage service we could just implement our own coverage target check as we have all the data already uploaded as artifacts. We could add a comment on each pull request pointing to the artifact documents.
Workaround¶
Create a pull request from the main repo as we have a token there.