action #120175
closed[sporadic]The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 size:M
Description
Observation¶
The CI badge in
https://github.com/os-autoinst/os-autoinst shows failed
Acceptance criteria¶
- AC1: The CI badge shows passed consistently
Suggestions¶
- Follow the link behind the badge, review CI failures, fix them
- Looks like the codecov stuff is broken and might be behind the errors
- Investigate retry in the codecov client
Updated by okurz almost 2 years ago
- Copied from action #119464: The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - cause: long commit msg size:M added
Updated by dheidler almost 2 years ago
- Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed to The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - gitlab <--> codecov api 404
Updated by tinita almost 2 years ago
I see such codecov failures regularly (like every few days), it seems to be a problem on their side
Updated by tinita almost 2 years ago
- Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - gitlab <--> codecov api 404 to The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404
Updated by livdywan almost 2 years ago
- Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 to [sporadic]The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by kraih almost 2 years ago
- Status changed from Workable to In Progress
Green badge today. Lets investigate a bit.
Updated by kraih almost 2 years ago
This looks like one such case from a few weeks ago: https://github.com/os-autoinst/os-autoinst/actions/runs/3144719216/jobs/5111084343
And two more for comparison:
Updated by kraih almost 2 years ago
Expected log output is:
[2022-11-10T19:54:15.937Z] ['info'] https://app.codecov.io/github/os-autoinst/os-autoinst/commit/7a0df628516146e83b57a178596abb47310e4fa7
https://storage.googleapis.com/codecov/v4/raw/2022-11-10/9E49A440727FB10136A3C5D4ED50CBD3/7a0df628516146e83b57a178596abb47310e4fa7/9776b8c2-afa1-48b0-8063-46d57d6f0d74.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221110%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221110T195415Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ce9c0ee2d1c851195935876baa1366a368045044b5dab55fd8254529a572094c
[2022-11-10T19:54:15.938Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2022-11-10/9E49A440727FB10136A3C5D4ED50CBD3/7a0df628516146e83b57a178596abb47310e4fa7/9776b8c2-afa1-48b0-8063-46d57d6f0d74.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221110%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221110T195415Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ce9c0ee2d1c851195935876baa1366a368045044b5dab55fd8254529a572094c
[2022-11-10T19:54:15.938Z] ['info'] Uploading...
[2022-11-10T19:54:16.169Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/os-autoinst/os-autoinst/commit/7a0df628516146e83b57a178596abb47310e4fa7"}
[2022-11-10T19:54:16.169Z] ['verbose'] End of uploader: 1214 milliseconds
Actual log output for the failing ones is:
[2022-10-04T14:39:35.375Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
[2022-10-04T14:39:35.376Z] ['verbose'] The error stack is: Error: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
at main (/snapshot/repo/dist/src/index.js)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-10-04T14:39:35.376Z] ['verbose'] End of uploader: 617 milliseconds
Error: Codecov: Failed to properly upload: The process '/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov' failed with exit code 255
So there's clearly an issue with the Codecov API where it does not return the requested upload URL. I'll check if it is possible to retry that step.
Updated by kraih almost 2 years ago
Noticed that we are not using the latest version of codecov/codecov-action
, made a PR to change that, in case that is part of the problem: https://github.com/os-autoinst/os-autoinst/pull/2203
Updated by kraih almost 2 years ago
If the migration from v2 to v3 doesn't help we do have other options as well:
- Retry the codecov action with a retry action
- Include a
CODECOV_TOKEN
in our config, even if it is not supposed to be required for a public GitHub repo (might sidestep the uploader bug) - Set
fail_ci_if_error: false
and ignore failed uploads
Updated by openqa_review almost 2 years ago
- Due date set to 2022-11-29
Setting due date based on mean cycle time of SUSE QE Tools
Updated by kraih almost 2 years ago
We just had a merge commit with the v3 action fail in the same way again, so that did not make a difference: https://github.com/os-autoinst/os-autoinst/actions/runs/3463073061/jobs/5782801673
Updated by kraih almost 2 years ago
Opened a new PR for the upload token solution: https://github.com/os-autoinst/os-autoinst/pull/2205
Updated by kraih almost 2 years ago
Interesting new datapoint. External branchaes can't use the token, since it is a repo secret. That means they fall back to the no token default for public repos and can experience the same problem. This has just been observed in: https://github.com/os-autoinst/os-autoinst/actions/runs/3471060371/jobs/5800053830 Rerun of the failed job passed: https://github.com/os-autoinst/os-autoinst/actions/runs/3471060371/jobs/5800657348
Updated by kraih almost 2 years ago
If the token actually works to prevent the problem, we could switch the workflow trigger from pull_request
to pull_request_target
. That would grant secret access. Security implications will have to be considered though. Even if the codecov token is not very critical at all, there might be other tokens in the future. https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
Updated by okurz almost 2 years ago
https://github.com/os-autoinst/os-autoinst shows a green badge right now. I assume this could be set to "Feedback" and awaiting results over the next days?
Updated by kraih almost 2 years ago
okurz wrote:
https://github.com/os-autoinst/os-autoinst shows a green badge right now. I assume this could be set to "Feedback" and awaiting results over the next days?
Yes, i'm keeping an eye on the badge for a few days now. But i thing the configured token does have a good chance to sidestep the issue. At least for the master branch.
Updated by kraih almost 2 years ago
- Status changed from In Progress to Feedback
Updated by kraih almost 2 years ago
Still looks ok for the master branch. Will wait a few more days before calling this resolved though.
Updated by livdywan almost 2 years ago
kraih wrote:
Still looks ok for the master branch. Will wait a few more days before calling this resolved though.
Seems like I just hit this on one of my branches: https://github.com/os-autoinst/os-autoinst/actions/runs/3515096990/jobs/5889928669
Updated by kraih almost 2 years ago
cdywan wrote:
Seems like I just hit this on one of my branches: https://github.com/os-autoinst/os-autoinst/actions/runs/3515096990/jobs/5889928669
Yea, looks like master is fine and we only see it sporadically in other branches now.
Updated by kraih almost 2 years ago
- Related to action #120843: [sporadic] Codecov upload errors for branches other than master - github <--> codecov api 404 added
Updated by kraih almost 2 years ago
Since this is only about the badge for the master branch, i consider this ticket resolved. But i've made a followup ticket for branches outside the os-autoinst
org. #120843