Project

General

Profile

Actions

action #120175

closed

[sporadic]The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 size:M

Added by okurz almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2022-10-26
Due date:
2022-11-29
% Done:

0%

Estimated time:

Description

Observation

The CI badge in
https://github.com/os-autoinst/os-autoinst shows failed

Acceptance criteria

  • AC1: The CI badge shows passed consistently

Suggestions

  • Follow the link behind the badge, review CI failures, fix them
  • Looks like the codecov stuff is broken and might be behind the errors
  • Investigate retry in the codecov client

Related issues 2 (1 open1 closed)

Related to openQA Project - action #120843: [sporadic] Codecov upload errors for branches other than master - github <--> codecov api 404New2022-11-22

Actions
Copied from openQA Project - action #119464: The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - cause: long commit msg size:MResolveddheidler2022-10-26

Actions
Actions #1

Updated by okurz almost 2 years ago

  • Copied from action #119464: The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - cause: long commit msg size:M added
Actions #2

Updated by dheidler almost 2 years ago

  • Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed to The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - gitlab <--> codecov api 404
Actions #3

Updated by tinita almost 2 years ago

I see such codecov failures regularly (like every few days), it seems to be a problem on their side

Actions #4

Updated by tinita almost 2 years ago

  • Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - gitlab <--> codecov api 404 to The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404
Actions #5

Updated by livdywan almost 2 years ago

  • Subject changed from The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 to [sporadic]The CI badge in https://github.com/os-autoinst/os-autoinst shows failed - github <--> codecov api 404 size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #6

Updated by kraih almost 2 years ago

  • Assignee set to kraih
Actions #7

Updated by kraih almost 2 years ago

  • Status changed from Workable to In Progress

Green badge today. Lets investigate a bit.

Actions #9

Updated by kraih almost 2 years ago

Expected log output is:

[2022-11-10T19:54:15.937Z] ['info'] https://app.codecov.io/github/os-autoinst/os-autoinst/commit/7a0df628516146e83b57a178596abb47310e4fa7
https://storage.googleapis.com/codecov/v4/raw/2022-11-10/9E49A440727FB10136A3C5D4ED50CBD3/7a0df628516146e83b57a178596abb47310e4fa7/9776b8c2-afa1-48b0-8063-46d57d6f0d74.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221110%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221110T195415Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ce9c0ee2d1c851195935876baa1366a368045044b5dab55fd8254529a572094c
[2022-11-10T19:54:15.938Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2022-11-10/9E49A440727FB10136A3C5D4ED50CBD3/7a0df628516146e83b57a178596abb47310e4fa7/9776b8c2-afa1-48b0-8063-46d57d6f0d74.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20221110%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20221110T195415Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ce9c0ee2d1c851195935876baa1366a368045044b5dab55fd8254529a572094c
[2022-11-10T19:54:15.938Z] ['info'] Uploading...
[2022-11-10T19:54:16.169Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/os-autoinst/os-autoinst/commit/7a0df628516146e83b57a178596abb47310e4fa7"}
[2022-11-10T19:54:16.169Z] ['verbose'] End of uploader: 1214 milliseconds

Actual log output for the failing ones is:

[2022-10-04T14:39:35.375Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
[2022-10-04T14:39:35.376Z] ['verbose'] The error stack is: Error: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
    at main (/snapshot/repo/dist/src/index.js)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-10-04T14:39:35.376Z] ['verbose'] End of uploader: 617 milliseconds
Error: Codecov: Failed to properly upload: The process '/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov' failed with exit code 255

So there's clearly an issue with the Codecov API where it does not return the requested upload URL. I'll check if it is possible to retry that step.

Actions #10

Updated by kraih almost 2 years ago

Noticed that we are not using the latest version of codecov/codecov-action, made a PR to change that, in case that is part of the problem: https://github.com/os-autoinst/os-autoinst/pull/2203

Actions #11

Updated by kraih almost 2 years ago

If the migration from v2 to v3 doesn't help we do have other options as well:

  • Retry the codecov action with a retry action
  • Include a CODECOV_TOKEN in our config, even if it is not supposed to be required for a public GitHub repo (might sidestep the uploader bug)
  • Set fail_ci_if_error: false and ignore failed uploads
Actions #12

Updated by openqa_review almost 2 years ago

  • Due date set to 2022-11-29

Setting due date based on mean cycle time of SUSE QE Tools

Actions #13

Updated by kraih almost 2 years ago

We just had a merge commit with the v3 action fail in the same way again, so that did not make a difference: https://github.com/os-autoinst/os-autoinst/actions/runs/3463073061/jobs/5782801673

Actions #14

Updated by kraih almost 2 years ago

Opened a new PR for the upload token solution: https://github.com/os-autoinst/os-autoinst/pull/2205

Actions #15

Updated by kraih almost 2 years ago

Interesting new datapoint. External branchaes can't use the token, since it is a repo secret. That means they fall back to the no token default for public repos and can experience the same problem. This has just been observed in: https://github.com/os-autoinst/os-autoinst/actions/runs/3471060371/jobs/5800053830 Rerun of the failed job passed: https://github.com/os-autoinst/os-autoinst/actions/runs/3471060371/jobs/5800657348

Actions #16

Updated by kraih almost 2 years ago

If the token actually works to prevent the problem, we could switch the workflow trigger from pull_request to pull_request_target. That would grant secret access. Security implications will have to be considered though. Even if the codecov token is not very critical at all, there might be other tokens in the future. https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

Actions #17

Updated by okurz almost 2 years ago

https://github.com/os-autoinst/os-autoinst shows a green badge right now. I assume this could be set to "Feedback" and awaiting results over the next days?

Actions #18

Updated by kraih almost 2 years ago

okurz wrote:

https://github.com/os-autoinst/os-autoinst shows a green badge right now. I assume this could be set to "Feedback" and awaiting results over the next days?

Yes, i'm keeping an eye on the badge for a few days now. But i thing the configured token does have a good chance to sidestep the issue. At least for the master branch.

Actions #19

Updated by kraih almost 2 years ago

  • Status changed from In Progress to Feedback
Actions #20

Updated by kraih almost 2 years ago

Still looks ok for the master branch. Will wait a few more days before calling this resolved though.

Actions #21

Updated by livdywan almost 2 years ago

kraih wrote:

Still looks ok for the master branch. Will wait a few more days before calling this resolved though.

Seems like I just hit this on one of my branches: https://github.com/os-autoinst/os-autoinst/actions/runs/3515096990/jobs/5889928669

Actions #22

Updated by kraih almost 2 years ago

cdywan wrote:

Seems like I just hit this on one of my branches: https://github.com/os-autoinst/os-autoinst/actions/runs/3515096990/jobs/5889928669

Yea, looks like master is fine and we only see it sporadically in other branches now.

Actions #23

Updated by kraih almost 2 years ago

  • Related to action #120843: [sporadic] Codecov upload errors for branches other than master - github <--> codecov api 404 added
Actions #24

Updated by kraih almost 2 years ago

Since this is only about the badge for the master branch, i consider this ticket resolved. But i've made a followup ticket for branches outside the os-autoinst org. #120843

Actions #25

Updated by kraih almost 2 years ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF