Project

General

Profile

action #120175

[sporadic]The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 size:M

Added by okurz 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2022-10-26
Due date:
2022-11-29
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

The CI badge in
https://github.com/os-autoinst/os-autoinst shows failed

Acceptance criteria

  • AC1: The CI badge shows passed consistently

Suggestions

  • Follow the link behind the badge, review CI failures, fix them
  • Looks like the codecov stuff is broken and might be behind the errors
  • Investigate retry in the codecov client

Related issues

Related to openQA Project - action #120843: [sporadic] Codecov upload errors for branches other than master - github <--> codecov api 404New2022-11-22

Copied from openQA Project - action #119464: The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - cause: long commit msg size:MResolved2022-10-26

History

#1 Updated by okurz 3 months ago

  • Copied from action #119464: The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - cause: long commit msg size:M added

#2 Updated by dheidler 3 months ago

  • Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed to The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - gitlab <--> codecov api 404

#3 Updated by tinita 3 months ago

I see such codecov failures regularly (like every few days), it seems to be a problem on their side

#4 Updated by tinita 3 months ago

  • Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - gitlab <--> codecov api 404 to The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404

#5 Updated by cdywan 3 months ago

  • Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 to [sporadic]The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 size:M
  • Description updated (diff)
  • Status changed from New to Workable

#6 Updated by kraih 3 months ago

  • Assignee set to kraih

#7 Updated by kraih 3 months ago

  • Status changed from Workable to In Progress

Green badge today. Lets investigate a bit.

#9 Updated by kraih 3 months ago

Expected log output is:

[2022-11-10T19:54:15.937Z] ['info'] https://app.codecov.io/github/os-autoinst/os-autoinst/commit/7a0df628516146e83b57a178596abb47310e4fa7
https://storage.googleapis.com/codecov/v4/raw/2022-11-10/9E49A440727FB10136A3C5D4ED50CBD3/7a0df628516146e83b57a178596abb47310e4fa7/9776b8c2-afa1-48b0-8063-46d57d6f0d74.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221110%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221110T195415Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ce9c0ee2d1c851195935876baa1366a368045044b5dab55fd8254529a572094c
[2022-11-10T19:54:15.938Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2022-11-10/9E49A440727FB10136A3C5D4ED50CBD3/7a0df628516146e83b57a178596abb47310e4fa7/9776b8c2-afa1-48b0-8063-46d57d6f0d74.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221110%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221110T195415Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ce9c0ee2d1c851195935876baa1366a368045044b5dab55fd8254529a572094c
[2022-11-10T19:54:15.938Z] ['info'] Uploading...
[2022-11-10T19:54:16.169Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/os-autoinst/os-autoinst/commit/7a0df628516146e83b57a178596abb47310e4fa7"}
[2022-11-10T19:54:16.169Z] ['verbose'] End of uploader: 1214 milliseconds

Actual log output for the failing ones is:

[2022-10-04T14:39:35.375Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
[2022-10-04T14:39:35.376Z] ['verbose'] The error stack is: Error: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
    at main (/snapshot/repo/dist/src/index.js)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-10-04T14:39:35.376Z] ['verbose'] End of uploader: 617 milliseconds
Error: Codecov: Failed to properly upload: The process '/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov' failed with exit code 255

So there's clearly an issue with the Codecov API where it does not return the requested upload URL. I'll check if it is possible to retry that step.

#10 Updated by kraih 3 months ago

Noticed that we are not using the latest version of codecov/codecov-action, made a PR to change that, in case that is part of the problem: https://github.com/os-autoinst/os-autoinst/pull/2203

#11 Updated by kraih 3 months ago

If the migration from v2 to v3 doesn't help we do have other options as well:

  • Retry the codecov action with a retry action
  • Include a CODECOV_TOKEN in our config, even if it is not supposed to be required for a public GitHub repo (might sidestep the uploader bug)
  • Set fail_ci_if_error: false and ignore failed uploads

#12 Updated by openqa_review 3 months ago

  • Due date set to 2022-11-29

Setting due date based on mean cycle time of SUSE QE Tools

#13 Updated by kraih 3 months ago

We just had a merge commit with the v3 action fail in the same way again, so that did not make a difference: https://github.com/os-autoinst/os-autoinst/actions/runs/3463073061/jobs/5782801673

#14 Updated by kraih 3 months ago

Opened a new PR for the upload token solution: https://github.com/os-autoinst/os-autoinst/pull/2205

#15 Updated by kraih 3 months ago

Interesting new datapoint. External branchaes can't use the token, since it is a repo secret. That means they fall back to the no token default for public repos and can experience the same problem. This has just been observed in: https://github.com/os-autoinst/os-autoinst/actions/runs/3471060371/jobs/5800053830 Rerun of the failed job passed: https://github.com/os-autoinst/os-autoinst/actions/runs/3471060371/jobs/5800657348

#16 Updated by kraih 3 months ago

If the token actually works to prevent the problem, we could switch the workflow trigger from pull_request to pull_request_target. That would grant secret access. Security implications will have to be considered though. Even if the codecov token is not very critical at all, there might be other tokens in the future. https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

#17 Updated by okurz 3 months ago

https://github.com/os-autoinst/os-autoinst shows a green badge right now. I assume this could be set to "Feedback" and awaiting results over the next days?

#18 Updated by kraih 3 months ago

okurz wrote:

https://github.com/os-autoinst/os-autoinst shows a green badge right now. I assume this could be set to "Feedback" and awaiting results over the next days?

Yes, i'm keeping an eye on the badge for a few days now. But i thing the configured token does have a good chance to sidestep the issue. At least for the master branch.

#19 Updated by kraih 3 months ago

  • Status changed from In Progress to Feedback

#20 Updated by kraih 3 months ago

Still looks ok for the master branch. Will wait a few more days before calling this resolved though.

#21 Updated by cdywan 3 months ago

kraih wrote:

Still looks ok for the master branch. Will wait a few more days before calling this resolved though.

Seems like I just hit this on one of my branches: https://github.com/os-autoinst/os-autoinst/actions/runs/3515096990/jobs/5889928669

#22 Updated by kraih 3 months ago

cdywan wrote:

Seems like I just hit this on one of my branches: https://github.com/os-autoinst/os-autoinst/actions/runs/3515096990/jobs/5889928669

Yea, looks like master is fine and we only see it sporadically in other branches now.

#23 Updated by kraih 3 months ago

  • Related to action #120843: [sporadic] Codecov upload errors for branches other than master - github <--> codecov api 404 added

#24 Updated by kraih 3 months ago

Since this is only about the badge for the master branch, i consider this ticket resolved. But i've made a followup ticket for branches outside the os-autoinst org. #120843

#25 Updated by kraih 3 months ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF