QA » openQA Project

I see such codecov failures regularly (like every few days), it seems to be a problem on their side

This looks like one such case from a few weeks ago: https://github.com/os-autoinst/os-autoinst/actions/runs/3144719216/jobs/5111084343

And two more for comparison:

• https://github.com/os-autoinst/os-autoinst/actions/runs/3182909261/jobs/5189465156
• https://github.com/os-autoinst/os-autoinst/actions/runs/3227670203/jobs/5282779980

Expected log output is:

[2022-11-10T19:54:15.937Z] ['info'] https://app.codecov.io/github/os-autoinst/os-autoinst/commit/7a0df628516146e83b57a178596abb47310e4fa7
https://storage.googleapis.com/codecov/v4/raw/2022-11-10/9E49A440727FB10136A3C5D4ED50CBD3/7a0df628516146e83b57a178596abb47310e4fa7/9776b8c2-afa1-48b0-8063-46d57d6f0d74.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221110%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221110T195415Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ce9c0ee2d1c851195935876baa1366a368045044b5dab55fd8254529a572094c
[2022-11-10T19:54:15.938Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2022-11-10/9E49A440727FB10136A3C5D4ED50CBD3/7a0df628516146e83b57a178596abb47310e4fa7/9776b8c2-afa1-48b0-8063-46d57d6f0d74.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221110%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221110T195415Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ce9c0ee2d1c851195935876baa1366a368045044b5dab55fd8254529a572094c
[2022-11-10T19:54:15.938Z] ['info'] Uploading...
[2022-11-10T19:54:16.169Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/os-autoinst/os-autoinst/commit/7a0df628516146e83b57a178596abb47310e4fa7"}
[2022-11-10T19:54:16.169Z] ['verbose'] End of uploader: 1214 milliseconds

Actual log output for the failing ones is:

[2022-10-04T14:39:35.375Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
[2022-10-04T14:39:35.376Z] ['verbose'] The error stack is: Error: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
    at main (/snapshot/repo/dist/src/index.js)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-10-04T14:39:35.376Z] ['verbose'] End of uploader: 617 milliseconds
Error: Codecov: Failed to properly upload: The process '/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov' failed with exit code 255

So there's clearly an issue with the Codecov API where it does not return the requested upload URL. I'll check if it is possible to retry that step.

Noticed that we are not using the latest version of codecov/codecov-action, made a PR to change that, in case that is part of the problem: https://github.com/os-autoinst/os-autoinst/pull/2203

If the migration from v2 to v3 doesn't help we do have other options as well:

• Retry the codecov action with a retry action
• Include a CODECOV_TOKEN in our config, even if it is not supposed to be required for a public GitHub repo (might sidestep the uploader bug)
• Set fail_ci_if_error: false and ignore failed uploads

We just had a merge commit with the v3 action fail in the same way again, so that did not make a difference: https://github.com/os-autoinst/os-autoinst/actions/runs/3463073061/jobs/5782801673

Opened a new PR for the upload token solution: https://github.com/os-autoinst/os-autoinst/pull/2205

Interesting new datapoint. External branchaes can't use the token, since it is a repo secret. That means they fall back to the no token default for public repos and can experience the same problem. This has just been observed in: https://github.com/os-autoinst/os-autoinst/actions/runs/3471060371/jobs/5800053830 Rerun of the failed job passed: https://github.com/os-autoinst/os-autoinst/actions/runs/3471060371/jobs/5800657348

If the token actually works to prevent the problem, we could switch the workflow trigger from pull_request to pull_request_target. That would grant secret access. Security implications will have to be considered though. Even if the codecov token is not very critical at all, there might be other tokens in the future. https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

https://github.com/os-autoinst/os-autoinst shows a green badge right now. I assume this could be set to "Feedback" and awaiting results over the next days?

okurz wrote:

https://github.com/os-autoinst/os-autoinst shows a green badge right now. I assume this could be set to "Feedback" and awaiting results over the next days?

Yes, i'm keeping an eye on the badge for a few days now. But i thing the configured token does have a good chance to sidestep the issue. At least for the master branch.

Still looks ok for the master branch. Will wait a few more days before calling this resolved though.

kraih wrote:

Still looks ok for the master branch. Will wait a few more days before calling this resolved though.

Seems like I just hit this on one of my branches: https://github.com/os-autoinst/os-autoinst/actions/runs/3515096990/jobs/5889928669

cdywan wrote:

Seems like I just hit this on one of my branches: https://github.com/os-autoinst/os-autoinst/actions/runs/3515096990/jobs/5889928669

Yea, looks like master is fine and we only see it sporadically in other branches now.

Since this is only about the badge for the master branch, i consider this ticket resolved. But i've made a followup ticket for branches outside the os-autoinst org. #120843