Project

General

Profile

Actions

action #128129

closed

codecov checks missing in pull requests size:M

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2023-04-21
Due date:
% Done:

0%

Estimated time:

Description

Observation

It seems multiple, maybe all, pull requests don't have codecov information. https://github.com/os-autoinst/openQA/pull/5090#issuecomment-1517387110 is one example. It seems only openQA fork-based pull requests are affected, not os-autoinst and also not pull requests from within the main repo.

OBS is also affected since some days, see their upstream report on https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266 . Problems likely started for us at the same time so either something on the side of codecov or circleCI or github changed.

Specific details from failing jobs:
https://app.circleci.com/pipelines/github/os-autoinst/openQA/11317/workflows/32a8f778-4ac3-429f-b4bb-6efc15a7cd15/jobs/105794?invite=true#step-112-43

[2023-04-21T09:17:36.470Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.5.0&token=*******&branch=pull%2F5088&build=105794&build_url=https%3A%2F%2Fcircleci.com%2Fgh%2Fos-autoinst%2FopenQA%2F105794&commit=a9a899bbdb18238728d38783e3ed47f2880a55b7&job=0&pr=5088&service=circleci&slug=os-autoinst%2FopenQA&name=&tag=&flags=&parent=
[2023-04-21T09:17:36.839Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Commit sha does not match Circle build. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
CircleCI received exit code 0

We have a circleCI token in the main repo but that is not accessible to forks. For os-autoinst fork-based PRs without access to a token also still seem to work where we use the github action that works.

Steps to reproduce

Likely any pull request created will not have codecov check results recorded

Expected result

  1. Pull requests for os-autoinst+openQA should have code coverage information
  2. PRs like "dependency cron" are merged automatically by mergify again

https://app.circleci.com/pipelines/github/os-autoinst/openQA/11371/workflows/d84e5d02-061e-4679-acdb-e2950b917ec8/jobs/106294 is a working example from a recent origin-based PR after the problem happened.

Suggestions

  • DONE Research about the problem if others are hit as well
  • DONE Try to reproduce the problem
  • Different alternatives we can follow:
1. Continue investigation to find a workaround as https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266 is not progressing. Maybe we find other github projects also using circleCI+codecov and they might have found better workarounds

 * For this trying with "circleCI orb": https://github.com/os-autoinst/openQA/pull/5110

2. Enable the use of codeclimate additionally to codecov, same as OBS did. Maybe also coveralls

3. Migrate from circleCI to github action – at least to crosscheck

Further details

See #127688, maybe related? - most likely not, see #12812#note-1

Out of scope

Instead of relying on any external coverage service we could just implement our own coverage target check as we have all the data already uploaded as artifacts. We could add a comment on each pull request pointing to the artifact documents.

Workaround

Create a pull request from the main repo as we have a token there.

Actions #1

Updated by mkittler over 1 year ago

  • Subject changed from codecov checks missing in pull requests - related to change to 2FA? to codecov checks missing in pull requests

Note that I have not enabled 2FA auth for the os-autoinst organization yet. The only bot account where I've enabled 2FA so far is "openqa-git-sync" but it shouldn't have anything to do with codecov. So this shouldn't be related at all to #127688.

Actions #2

Updated by mkittler over 1 year ago

  • Description updated (diff)
Actions #3

Updated by tinita over 1 year ago

https://app.circleci.com/pipelines/github/os-autoinst/openQA/11317/workflows/32a8f778-4ac3-429f-b4bb-6efc15a7cd15/jobs/105794?invite=true#step-112-43

[2023-04-21T09:17:36.470Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.5.0&token=*******&branch=pull%2F5088&build=105794&build_url=https%3A%2F%2Fcircleci.com%2Fgh%2Fos-autoinst%2FopenQA%2F105794&commit=a9a899bbdb18238728d38783e3ed47f2880a55b7&job=0&pr=5088&service=circleci&slug=os-autoinst%2FopenQA&name=&tag=&flags=&parent=
[2023-04-21T09:17:36.839Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Commit sha does not match Circle build. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
CircleCI received exit code 0

Possible bug: https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266

Actions #4

Updated by mkittler over 1 year ago

  • Assignee set to mkittler
Actions #5

Updated by mkittler over 1 year ago

Right, I found it was well. One additional problem is definitely that the CircleCI step still appears successful.

Actions #6

Updated by mkittler over 1 year ago

  • Status changed from New to In Progress

I have added the CODECOV_TOKEN as CircleCI env variable as suggested in the thread Tina linked. Let's see whether it helps. We already specify that variable in os-autoinst to avoid sporadic issues. That would explain why we haven't seen the issue there.

Actions #7

Updated by mkittler over 1 year ago

PR for "One additional problem is definitely that the CircleCI step still appears successful.": https://github.com/os-autoinst/openQA/pull/5091

Actions #8

Updated by mkittler over 1 year ago

Passing/setting the env variable this way didn't work.

Supposedly os-autoinst only works because it uses https://github.com/codecov/codecov-action but I'm not sure how we could use that with CircleCI.

Actions #9

Updated by openqa_review over 1 year ago

  • Due date set to 2023-05-06

Setting due date based on mean cycle time of SUSE QE Tools

Actions #10

Updated by mkittler over 1 year ago

  • Status changed from In Progress to New

It is basically blocked by https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266/8?u=martchus.

However, if someone knows how to set environment variables containing secrets for our CircleCI jobs (also covering runs for branches on forks), feel free to pick this ticket up and set CODECOV_TOKEN as a workaround. (You can find the token on https://app.codecov.io/gh/os-autoinst/openQA/settings.)

Actions #11

Updated by mkittler over 1 year ago

  • Assignee deleted (mkittler)
Actions #12

Updated by mkittler over 1 year ago

  • Status changed from New to In Progress
  • Assignee set to mkittler

I found another place where I can add environment variables in CircleCI so I'll try that.

EDIT: Added the variable on https://app.circleci.com/settings/organization/github/os-autoinst/contexts/3a035914-de09-4dcf-bd17-f664bc3e2c77 and force-pushed on https://github.com/os-autoinst/openQA/pull/5091 to check whether it works.

Actions #13

Updated by mkittler over 1 year ago

  • Status changed from In Progress to Blocked

Looks like this didn't work (also not with https://github.com/os-autoinst/openQA/pull/5091/commits/01f9995e52fd902f195e4cc6a3ee73a97a35eace). So I'm running out of ideas for setting the env variable. Any ideas are welcome.

I suppose I'll keep the ticket assigned to myself and block it on https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266/8?u=martchus.

Actions #14

Updated by livdywan over 1 year ago

For now, the workaround is to specify -C $CIRCLE_SHA1. I'll update here as we work through the issue.

Did you try this somewhere, too?

Also, I wonder what also strongly encourage you to switch to the actual uploader is about. Is there yet another tool we should consider switching to? Or are we already using it?

Actions #15

Updated by mkittler over 1 year ago

Where does this quote come from? I'm not really sure what it means without context.

I'm also not sure what they mean with "actual uploader". Supposedly the GitHub action that we also already use in os-autoinst (and that indeed continues to work without token). However, in openQA we use CircleCI so I don't think we can easily use the GitHub action. (So we use some binary from codecov that is downloaded via curl in the CircleCI job.)

Actions #16

Updated by mkittler over 1 year ago

Note that setting the token works if one creates a PR from a branch in the main repository (and one's fork; like this: https://github.com/os-autoinst/openQA/pull/5105). That would at least be a workaround for team members with push access to the main repo.

Actions #17

Updated by livdywan over 1 year ago

mkittler wrote:

Where does this quote come from? I'm not really sure what it means without context.

I'm also not sure what they mean with "actual uploader". Supposedly the GitHub action that we also already use in os-autoinst (and that indeed continues to work without token). However, in openQA we use CircleCI so I don't think we can easily use the GitHub action. (So we use some binary from codecov that is downloaded via curl in the CircleCI job.)

I was quoting from https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266/8?u=martchus which you mentioned before. Sorry if that wasn't clear.

https://uploader.codecov.io/linux/latest is what we use, right? And there is also the orb but that uses the same uploader in the implementation so it probably wouldn't help us.

So I guess that is not helpful anyway.

Actions #18

Updated by mkittler over 1 year ago

Yes, it looks like the orb is just using the same downloader (https://github.com/codecov/codecov-circleci-orb/blob/master/src/scripts/download.sh).

Actions #19

Updated by okurz over 1 year ago

We looked into this problem together after the team's daily. We tried to reproduce the problem manually with downloading the codecov uploader binary as well as setting the same environment variables as circleCI does. Comparing to os-autoinst we found that os-autoinst is using the same uploader internally but with command line parameters like -n -Q github-action-3.1.3 -Z -v …. Specifying the same still reproduces the problem. @mkittler added the -v in https://github.com/os-autoinst/openQA/pull/5091 . We should await what https://app.circleci.com/jobs/github/os-autoinst/openQA/106382 will say.

https://app.codecov.io/gh/openSUSE/open-build-service/pulls shows that OBS has not found a fix yet as well but they seem to have settled for codeclimate as alternative, see https://github.com/openSUSE/open-build-service/pull/14141 , while they still upload to codecov.

Different routes we can go:

  1. Continue investigation to find a workaround as https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266 is not progressing. Maybe we find other github projects also using circleCI+codecov and they might have found better workarounds

  2. Migrate from circleCI to github action – at least to crosscheck

  3. Enable the use of codeclimate additionally to codecov, same as OBS did

Actions #20

Updated by mkittler over 1 year ago

A few further remarks from today's session:


¹

[2023-05-02T09:31:23.041Z] ['verbose'] Start of uploader: 1683019883041...
[2023-05-02T09:31:23.042Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.5.0
[2023-05-02T09:31:23.046Z] ['info'] => Project root located at: /home/squamata/project
[2023-05-02T09:31:23.050Z] ['info'] -> No token specified or token is empty
[2023-05-02T09:31:23.050Z] ['verbose'] Start of network processing...
[2023-05-02T09:31:23.050Z] ['verbose'] Searching for files in /home/squamata/project
[2023-05-02T09:31:23.059Z] ['verbose'] coveragepy is not installed
[2023-05-02T09:31:23.059Z] ['info'] Searching for coverage files...
[2023-05-02T09:31:23.279Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-05-02T09:31:23.279Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-05-02T09:31:23.279Z] ['verbose'] Preparing to clean the following coverage paths: cover_db/codecov.json,t/data/openqa-trigger-from-obs/Proj1/.run_190701_143010_468.2/files_iso.lst,t/data/openqa-trigger-from-obs/Proj1/.run_last/files_iso.lst,t/data/openqa-trigger-from-obs/Proj1/.run_190703_143010_469.1/files_iso.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/Media1_ftp_ftp.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/files_iso.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/Media1_ftp_ftp.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/files_iso.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_191216_150610/files_iso.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_last/files_iso.lst
[2023-05-02T09:31:23.280Z] ['info'] => Found 10 possible coverage files:
  cover_db/codecov.json
  t/data/openqa-trigger-from-obs/Proj1/.run_190701_143010_468.2/files_iso.lst
  t/data/openqa-trigger-from-obs/Proj1/.run_last/files_iso.lst
  t/data/openqa-trigger-from-obs/Proj1/.run_190703_143010_469.1/files_iso.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/Media1_ftp_ftp.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/files_iso.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/Media1_ftp_ftp.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/files_iso.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_191216_150610/files_iso.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_last/files_iso.lst
[2023-05-02T09:31:23.280Z] ['verbose'] End of network processing
[2023-05-02T09:31:23.280Z] ['info'] Processing /home/squamata/project/cover_db/codecov.json...
[2023-05-02T09:31:23.282Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/Proj1/.run_190701_143010_468.2/files_iso.lst...
[2023-05-02T09:31:23.282Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/Proj1/.run_last/files_iso.lst...
[2023-05-02T09:31:23.282Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/Proj1/.run_190703_143010_469.1/files_iso.lst...
[2023-05-02T09:31:23.283Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/Media1_ftp_ftp.lst...
[2023-05-02T09:31:23.283Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/files_iso.lst...
[2023-05-02T09:31:23.283Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/Media1_ftp_ftp.lst...
[2023-05-02T09:31:23.284Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/files_iso.lst...
[2023-05-02T09:31:23.284Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_191216_150610/files_iso.lst...
[2023-05-02T09:31:23.284Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_last/files_iso.lst...
[2023-05-02T09:31:23.295Z] ['info'] Detected CircleCI as the CI provider.
[2023-05-02T09:31:23.295Z] ['verbose'] -> Using the following env variables:
[2023-05-02T09:31:23.296Z] ['verbose']      CI: true
[2023-05-02T09:31:23.296Z] ['verbose']      CIRCLECI: true
[2023-05-02T09:31:23.296Z] ['verbose'] Using the following upload parameters:
[2023-05-02T09:31:23.296Z] ['verbose'] branch
[2023-05-02T09:31:23.296Z] ['verbose'] build
[2023-05-02T09:31:23.296Z] ['verbose'] buildURL
[2023-05-02T09:31:23.296Z] ['verbose'] commit
[2023-05-02T09:31:23.296Z] ['verbose'] job
[2023-05-02T09:31:23.296Z] ['verbose'] pr
[2023-05-02T09:31:23.296Z] ['verbose'] service
[2023-05-02T09:31:23.296Z] ['verbose'] slug
[2023-05-02T09:31:23.296Z] ['verbose'] name
[2023-05-02T09:31:23.296Z] ['verbose'] tag
[2023-05-02T09:31:23.296Z] ['verbose'] flags
[2023-05-02T09:31:23.296Z] ['verbose'] parent
[2023-05-02T09:31:23.297Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.5.0&token=*******&branch=pull%2F5091&build=106382&build_url=https%3A%2F%2Fcircleci.com%2Fgh%2Fos-autoinst%2FopenQA%2F106382&commit=e8ab577ef2c33a6dc479b2d5ffc9069b3fb0f616&job=0&pr=5091&service=circleci&slug=os-autoinst%2FopenQA&name=&tag=&flags=&parent=
[2023-05-02T09:31:23.297Z] ['verbose'] Passed token was 0 characters long
[2023-05-02T09:31:23.297Z] ['verbose'] https://codecov.io/upload/v4?package=uploader-0.5.0&branch=pull%2F5091&build=106382&build_url=https%3A%2F%2Fcircleci.com%2Fgh%2Fos-autoinst%2FopenQA%2F106382&commit=e8ab577ef2c33a6dc479b2d5ffc9069b3fb0f616&job=0&pr=5091&service=circleci&slug=os-autoinst%2FopenQA&name=&tag=&flags=&parent=
        Content-Type: 'text/plain'
        Content-Encoding: 'gzip'
        X-Reduced-Redundancy: 'false'
[2023-05-02T09:31:23.722Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Commit sha does not match Circle build. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
[2023-05-02T09:31:23.723Z] ['verbose'] The error stack is: Error: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Commit sha does not match Circle build. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
    at main (/snapshot/repo/dist/src/index.js)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2023-05-02T09:31:23.723Z] ['verbose'] End of uploader: 682 milliseconds
Actions #21

Updated by okurz over 1 year ago

  • Due date changed from 2023-05-06 to 2023-05-12
  • Status changed from Blocked to In Progress
  • Assignee changed from mkittler to okurz
Actions #22

Updated by okurz over 1 year ago

  • Subject changed from codecov checks missing in pull requests to codecov checks missing in pull requests size:M
  • Description updated (diff)
Actions #23

Updated by okurz over 1 year ago

Trying to specify codecov.json file explicitly plus "extra args". In the latest commit also adding the SHA of the commit with

xtra_args: -v -Z -C $CIRCLE_SHA1

Specifying the git hash explicitly as https://discuss.circleci.com/t/different-commit-sha-makes-codecov-coverage-report-fail/22636 says that the computed hash might differ, similar from https://discuss.circleci.com/t/circleci-short-object-hash-does-not-match-gits/36010 . And https://community.codecov.com/t/codecov-uploads-from-azure-pipelines-are-failing-with-build-numbers-do-not-match/1100/13 suggests that one specifies the hash directly. https://community.codecov.com/t/codecov-uploads-from-azure-pipelines-are-failing-with-build-numbers-do-not-match/1100/47 has a lengthy description and mentions of fixes from codecov side but no more details what fixed it, only that in 2020 there was an actual issue server-side.

https://github.com/codecov/codecov-bash/issues/287#issuecomment-612522516 mentions an Azure specific problem that also got fixed by specifying -C explicitly.

EDIT: https://app.circleci.com/pipelines/github/os-autoinst/openQA/11403/workflows/06c21cab-7c66-49ee-a1d4-593c1f313ef0/jobs/106581 shows that the URL to codecov includes the correct git commit but failing with the same error. I also found a problem on os-autoinst where the CI job "ci/test" failed in https://github.com/os-autoinst/os-autoinst/actions/runs/4881705923/jobs/8710922631?pr=2311#step:4:97 with the same error. So likely the same sporadic error in pull requests that we already know. I assume retrying works there. So maybe we need to retry in other cases as well? That worked now for retrying one os-autoinst PR stuff.

We have explored alternatives like using codeclimate and coveralls. codeclimate does not support Perl code so not for us. And with coveralls in the end we also need a token also even for pull requests. So in the end we decided that we should just specify the token in for codecov. The token is as specified on https://app.codecov.io/gh/os-autoinst/openQA/settings . First I followed https://circleci.com/blog/making-code-coverage-easy-to-see-with-the-codecov-orb/ to specify the token for the orb but that failed with some syntax error so now specifying the token in codecov.yml

Actions #24

Updated by tinita over 1 year ago

https://build.opensuse.org/request/show/1084669 Submit package devel:languages:perl / perl-Mozilla-CA to package openSUSE:Factory / perl-Mozilla-CA
https://build.opensuse.org/request/show/1084671 Submit package devel:languages:perl / perl-Devel-Cover-Report-Coveralls to package openSUSE:Factory / perl-Devel-...-Coveralls

Actions #25

Updated by okurz over 1 year ago

  • Status changed from In Progress to Feedback

https://github.com/os-autoinst/openQA/pull/5110 merged, monitoring results from other PRs.

Actions #26

Updated by okurz over 1 year ago

  • Due date deleted (2023-05-12)
  • Status changed from Feedback to Resolved

https://github.com/os-autoinst/openQA/pull/5120 is a "Dependency cron" PR that was merged automatically so we are good

Actions

Also available in: Atom PDF