QA » openQA Project

https://app.circleci.com/pipelines/github/os-autoinst/openQA/11317/workflows/32a8f778-4ac3-429f-b4bb-6efc15a7cd15/jobs/105794?invite=true#step-112-43

[2023-04-21T09:17:36.470Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.5.0&token=*******&branch=pull%2F5088&build=105794&build_url=https%3A%2F%2Fcircleci.com%2Fgh%2Fos-autoinst%2FopenQA%2F105794&commit=a9a899bbdb18238728d38783e3ed47f2880a55b7&job=0&pr=5088&service=circleci&slug=os-autoinst%2FopenQA&name=&tag=&flags=&parent=
[2023-04-21T09:17:36.839Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Commit sha does not match Circle build. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
CircleCI received exit code 0

Possible bug: https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266

Right, I found it was well. One additional problem is definitely that the CircleCI step still appears successful.

PR for "One additional problem is definitely that the CircleCI step still appears successful.": https://github.com/os-autoinst/openQA/pull/5091

Passing/setting the env variable this way didn't work.

Supposedly os-autoinst only works because it uses https://github.com/codecov/codecov-action but I'm not sure how we could use that with CircleCI.

For now, the workaround is to specify -C $CIRCLE_SHA1. I'll update here as we work through the issue.

Did you try this somewhere, too?

Also, I wonder what also strongly encourage you to switch to the actual uploader is about. Is there yet another tool we should consider switching to? Or are we already using it?

Where does this quote come from? I'm not really sure what it means without context.

I'm also not sure what they mean with "actual uploader". Supposedly the GitHub action that we also already use in os-autoinst (and that indeed continues to work without token). However, in openQA we use CircleCI so I don't think we can easily use the GitHub action. (So we use some binary from codecov that is downloaded via curl in the CircleCI job.)

Note that setting the token works if one creates a PR from a branch in the main repository (and one's fork; like this: https://github.com/os-autoinst/openQA/pull/5105). That would at least be a workaround for team members with push access to the main repo.

mkittler wrote:

Where does this quote come from? I'm not really sure what it means without context.

I'm also not sure what they mean with "actual uploader". Supposedly the GitHub action that we also already use in os-autoinst (and that indeed continues to work without token). However, in openQA we use CircleCI so I don't think we can easily use the GitHub action. (So we use some binary from codecov that is downloaded via curl in the CircleCI job.)

I was quoting from https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266/8?u=martchus which you mentioned before. Sorry if that wasn't clear.

https://uploader.codecov.io/linux/latest is what we use, right? And there is also the orb but that uses the same uploader in the implementation so it probably wouldn't help us.

So I guess that is not helpful anyway.

Yes, it looks like the orb is just using the same downloader (https://github.com/codecov/codecov-circleci-orb/blob/master/src/scripts/download.sh).

We looked into this problem together after the team's daily. We tried to reproduce the problem manually with downloading the codecov uploader binary as well as setting the same environment variables as circleCI does. Comparing to os-autoinst we found that os-autoinst is using the same uploader internally but with command line parameters like -n -Q github-action-3.1.3 -Z -v …. Specifying the same still reproduces the problem. @mkittler added the -v in https://github.com/os-autoinst/openQA/pull/5091 . We should await what https://app.circleci.com/jobs/github/os-autoinst/openQA/106382 will say.

https://app.codecov.io/gh/openSUSE/open-build-service/pulls shows that OBS has not found a fix yet as well but they seem to have settled for codeclimate as alternative, see https://github.com/openSUSE/open-build-service/pull/14141 , while they still upload to codecov.

Different routes we can go:

1. Continue investigation to find a workaround as https://community.codecov.com/t/commit-sha-does-not-match-circle-build/4266 is not progressing. Maybe we find other github projects also using circleCI+codecov and they might have found better workarounds

   • For this trying with "circleCI orb": https://github.com/os-autoinst/openQA/pull/5110

2. Migrate from circleCI to github action – at least to crosscheck

3. Enable the use of codeclimate additionally to codecov, same as OBS did

A few further remarks from today's session:

• After adding -v in my PR we don't get that much more information. The message "coveragepy is not installed" is also present in the good case.
• Since the GitHub action is actually just using the binary after all, the difference is likely in CI-runner specific checking on the Codecov server-side or in the CI-runner specific client-side code:
  • https://github.com/codecov/uploader/blob/main/src/ci_providers/provider_githubactions.ts
  • https://github.com/codecov/uploader/blob/main/src/ci_providers/provider_circleci.ts - this file hasn't seen any relevant changes recently, though

¹

[2023-05-02T09:31:23.041Z] ['verbose'] Start of uploader: 1683019883041...
[2023-05-02T09:31:23.042Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.5.0
[2023-05-02T09:31:23.046Z] ['info'] => Project root located at: /home/squamata/project
[2023-05-02T09:31:23.050Z] ['info'] -> No token specified or token is empty
[2023-05-02T09:31:23.050Z] ['verbose'] Start of network processing...
[2023-05-02T09:31:23.050Z] ['verbose'] Searching for files in /home/squamata/project
[2023-05-02T09:31:23.059Z] ['verbose'] coveragepy is not installed
[2023-05-02T09:31:23.059Z] ['info'] Searching for coverage files...
[2023-05-02T09:31:23.279Z] ['info'] Warning: Some files located via search were excluded from upload.
[2023-05-02T09:31:23.279Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats
[2023-05-02T09:31:23.279Z] ['verbose'] Preparing to clean the following coverage paths: cover_db/codecov.json,t/data/openqa-trigger-from-obs/Proj1/.run_190701_143010_468.2/files_iso.lst,t/data/openqa-trigger-from-obs/Proj1/.run_last/files_iso.lst,t/data/openqa-trigger-from-obs/Proj1/.run_190703_143010_469.1/files_iso.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/Media1_ftp_ftp.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/files_iso.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/Media1_ftp_ftp.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/files_iso.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_191216_150610/files_iso.lst,t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_last/files_iso.lst
[2023-05-02T09:31:23.280Z] ['info'] => Found 10 possible coverage files:
  cover_db/codecov.json
  t/data/openqa-trigger-from-obs/Proj1/.run_190701_143010_468.2/files_iso.lst
  t/data/openqa-trigger-from-obs/Proj1/.run_last/files_iso.lst
  t/data/openqa-trigger-from-obs/Proj1/.run_190703_143010_469.1/files_iso.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/Media1_ftp_ftp.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/files_iso.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/Media1_ftp_ftp.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/files_iso.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_191216_150610/files_iso.lst
  t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_last/files_iso.lst
[2023-05-02T09:31:23.280Z] ['verbose'] End of network processing
[2023-05-02T09:31:23.280Z] ['info'] Processing /home/squamata/project/cover_db/codecov.json...
[2023-05-02T09:31:23.282Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/Proj1/.run_190701_143010_468.2/files_iso.lst...
[2023-05-02T09:31:23.282Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/Proj1/.run_last/files_iso.lst...
[2023-05-02T09:31:23.282Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/Proj1/.run_190703_143010_469.1/files_iso.lst...
[2023-05-02T09:31:23.283Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/Media1_ftp_ftp.lst...
[2023-05-02T09:31:23.283Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/files_iso.lst...
[2023-05-02T09:31:23.283Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/Media1_ftp_ftp.lst...
[2023-05-02T09:31:23.284Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_last/files_iso.lst...
[2023-05-02T09:31:23.284Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_191216_150610/files_iso.lst...
[2023-05-02T09:31:23.284Z] ['info'] Processing /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_last/files_iso.lst...
[2023-05-02T09:31:23.295Z] ['info'] Detected CircleCI as the CI provider.
[2023-05-02T09:31:23.295Z] ['verbose'] -> Using the following env variables:
[2023-05-02T09:31:23.296Z] ['verbose']      CI: true
[2023-05-02T09:31:23.296Z] ['verbose']      CIRCLECI: true
[2023-05-02T09:31:23.296Z] ['verbose'] Using the following upload parameters:
[2023-05-02T09:31:23.296Z] ['verbose'] branch
[2023-05-02T09:31:23.296Z] ['verbose'] build
[2023-05-02T09:31:23.296Z] ['verbose'] buildURL
[2023-05-02T09:31:23.296Z] ['verbose'] commit
[2023-05-02T09:31:23.296Z] ['verbose'] job
[2023-05-02T09:31:23.296Z] ['verbose'] pr
[2023-05-02T09:31:23.296Z] ['verbose'] service
[2023-05-02T09:31:23.296Z] ['verbose'] slug
[2023-05-02T09:31:23.296Z] ['verbose'] name
[2023-05-02T09:31:23.296Z] ['verbose'] tag
[2023-05-02T09:31:23.296Z] ['verbose'] flags
[2023-05-02T09:31:23.296Z] ['verbose'] parent
[2023-05-02T09:31:23.297Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.5.0&token=*******&branch=pull%2F5091&build=106382&build_url=https%3A%2F%2Fcircleci.com%2Fgh%2Fos-autoinst%2FopenQA%2F106382&commit=e8ab577ef2c33a6dc479b2d5ffc9069b3fb0f616&job=0&pr=5091&service=circleci&slug=os-autoinst%2FopenQA&name=&tag=&flags=&parent=
[2023-05-02T09:31:23.297Z] ['verbose'] Passed token was 0 characters long
[2023-05-02T09:31:23.297Z] ['verbose'] https://codecov.io/upload/v4?package=uploader-0.5.0&branch=pull%2F5091&build=106382&build_url=https%3A%2F%2Fcircleci.com%2Fgh%2Fos-autoinst%2FopenQA%2F106382&commit=e8ab577ef2c33a6dc479b2d5ffc9069b3fb0f616&job=0&pr=5091&service=circleci&slug=os-autoinst%2FopenQA&name=&tag=&flags=&parent=
        Content-Type: 'text/plain'
        Content-Encoding: 'gzip'
        X-Reduced-Redundancy: 'false'
[2023-05-02T09:31:23.722Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Commit sha does not match Circle build. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
[2023-05-02T09:31:23.723Z] ['verbose'] The error stack is: Error: Error uploading to https://codecov.io: Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Commit sha does not match Circle build. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
    at main (/snapshot/repo/dist/src/index.js)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2023-05-02T09:31:23.723Z] ['verbose'] End of uploader: 682 milliseconds

Trying to specify codecov.json file explicitly plus "extra args". In the latest commit also adding the SHA of the commit with

xtra_args: -v -Z -C $CIRCLE_SHA1

Specifying the git hash explicitly as https://discuss.circleci.com/t/different-commit-sha-makes-codecov-coverage-report-fail/22636 says that the computed hash might differ, similar from https://discuss.circleci.com/t/circleci-short-object-hash-does-not-match-gits/36010 . And https://community.codecov.com/t/codecov-uploads-from-azure-pipelines-are-failing-with-build-numbers-do-not-match/1100/13 suggests that one specifies the hash directly. https://community.codecov.com/t/codecov-uploads-from-azure-pipelines-are-failing-with-build-numbers-do-not-match/1100/47 has a lengthy description and mentions of fixes from codecov side but no more details what fixed it, only that in 2020 there was an actual issue server-side.

https://github.com/codecov/codecov-bash/issues/287#issuecomment-612522516 mentions an Azure specific problem that also got fixed by specifying -C explicitly.

EDIT: https://app.circleci.com/pipelines/github/os-autoinst/openQA/11403/workflows/06c21cab-7c66-49ee-a1d4-593c1f313ef0/jobs/106581 shows that the URL to codecov includes the correct git commit but failing with the same error. I also found a problem on os-autoinst where the CI job "ci/test" failed in https://github.com/os-autoinst/os-autoinst/actions/runs/4881705923/jobs/8710922631?pr=2311#step:4:97 with the same error. So likely the same sporadic error in pull requests that we already know. I assume retrying works there. So maybe we need to retry in other cases as well? That worked now for retrying one os-autoinst PR stuff.

We have explored alternatives like using codeclimate and coveralls. codeclimate does not support Perl code so not for us. And with coveralls in the end we also need a token also even for pull requests. So in the end we decided that we should just specify the token in for codecov. The token is as specified on https://app.codecov.io/gh/os-autoinst/openQA/settings . First I followed https://circleci.com/blog/making-code-coverage-easy-to-see-with-the-codecov-orb/ to specify the token for the orb but that failed with some syntax error so now specifying the token in codecov.yml

https://build.opensuse.org/request/show/1084669 Submit package devel:languages:perl / perl-Mozilla-CA to package openSUSE:Factory / perl-Mozilla-CA
https://build.opensuse.org/request/show/1084671 Submit package devel:languages:perl / perl-Devel-Cover-Report-Coveralls to package openSUSE:Factory / perl-Devel-...-Coveralls