action #163187
closedUploading coverage to Codecov via the CircleCI setup of openQA no longer works size:M
0%
Description
It runs into a 401 error, currently most of the time:
./"codecov" -v create-commit -t <redacted>
info - 2024-06-25 15:25:46,820 -- ci service found: circleci
debug - 2024-06-25 15:25:46,822 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-06-25 15:25:46,824 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-06-25 15:25:46,825 -- Loading config from /home/squamata/project/codecov.yml
debug - 2024-06-25 15:25:46,830 -- Starting create commit process --- {"commit_sha": "d100e336909c09e6cb8fe51e6c92488610634411", "parent_sha": null, "pr": "5723", "branch": "pull/5723", "slug": "os-autoinst/openQA", "token": "", "service": "github", "enterprise_url": null}
info - 2024-06-25 15:25:46,898 -- Process Commit creating complete
debug - 2024-06-25 15:25:46,899 -- Commit creating result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 401', params={}, description='{\"detail\":\"Invalid token header. No credentials provided.\"}'), warnings=[], status_code=401, text='{\"detail\":\"Invalid token header. No credentials provided.\"}')"}
error - 2024-06-25 15:25:46,899 -- Commit creating failed: {"detail":"Invalid token header. No credentials provided."}
./"codecov" -v create-report -t <redacted>
info - 2024-06-25 15:25:47,778 -- ci service found: circleci
debug - 2024-06-25 15:25:47,779 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-06-25 15:25:47,781 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-06-25 15:25:47,782 -- Loading config from /home/squamata/project/codecov.yml
debug - 2024-06-25 15:25:47,788 -- Starting create report process --- {"commit_sha": "d100e336909c09e6cb8fe51e6c92488610634411", "code": "default", "slug": "os-autoinst/openQA", "service": "github", "enterprise_url": null, "token": ""}
info - 2024-06-25 15:25:47,826 -- Process Report creating complete
debug - 2024-06-25 15:25:47,826 -- Report creating result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 401', params={}, description='{\"detail\":\"Invalid token header. No credentials provided.\"}'), warnings=[], status_code=401, text='{\"detail\":\"Invalid token header. No credentials provided.\"}')"}
error - 2024-06-25 15:25:47,826 -- Report creating failed: {"detail":"Invalid token header. No credentials provided."}
./codecov -v do-upload -t <redacted> -n "130467" -f cover_db/codecov.json
info - 2024-06-25 15:25:48,692 -- ci service found: circleci
debug - 2024-06-25 15:25:48,693 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-06-25 15:25:48,695 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-06-25 15:25:48,696 -- Loading config from /home/squamata/project/codecov.yml
debug - 2024-06-25 15:25:48,702 -- Starting upload processing --- {"branch": "pull/5723", "build_code": "130467", "build_url": "https://circleci.com/gh/os-autoinst/openQA/130467", "commit_sha": "d100e336909c09e6cb8fe51e6c92488610634411", "disable_file_fixes": false, "disable_search": false, "enterprise_url": null, "env_vars": {}, "files_search_exclude_folders": [], "files_search_explicitly_listed_files": ["cover_db/codecov.json"], "files_search_root_folder": "/home/squamata/project", "flags": [], "git_service": "github", "handle_no_reports_found": false, "job_code": "0", "name": "130467", "network_filter": null, "network_prefix": null, "network_root_folder": "/home/squamata/project", "plugin_names": ["xcode", "gcov", "pycoverage"], "pull_request_number": "5723", "report_code": "default", "slug": "os-autoinst/openQA", "token": "", "upload_file_type": "coverage"}
debug - 2024-06-25 15:25:48,702 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"]}
debug - 2024-06-25 15:25:48,702 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-06-25 15:25:48,702 -- Running xcode plugin...
warning - 2024-06-25 15:25:48,702 -- xcrun is not installed or can't be found.
debug - 2024-06-25 15:25:48,702 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-06-25 15:25:48,702 -- Running gcov plugin...
warning - 2024-06-25 15:25:48,822 -- No gcov data found.
debug - 2024-06-25 15:25:48,822 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
warning - 2024-06-25 15:25:48,822 -- coverage.py is not installed or can't be found.
debug - 2024-06-25 15:25:48,823 -- Collecting relevant files
info - 2024-06-25 15:25:49,135 -- Found 6 coverage files to report
info - 2024-06-25 15:25:49,135 -- > /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/Media1_ftp_ftp.lst
info - 2024-06-25 15:25:49,135 -- > /home/squamata/project/t/data/openqa-trigger-from-obs/Proj1/.run_190703_143010_469.1/files_iso.lst
info - 2024-06-25 15:25:49,135 -- > /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch2/.run_191216_150610/files_iso.lst
info - 2024-06-25 15:25:49,135 -- > /home/squamata/project/cover_db/codecov.json
info - 2024-06-25 15:25:49,135 -- > /home/squamata/project/t/data/openqa-trigger-from-obs/BatchedProj/Batch1/.run_191216_150610/files_iso.lst
info - 2024-06-25 15:25:49,135 -- > /home/squamata/project/t/data/openqa-trigger-from-obs/Proj1/.run_190701_143010_468.2/files_iso.lst
debug - 2024-06-25 15:25:49,163 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-06-25 15:25:49,173 -- Sending upload request to Codecov
info - 2024-06-25 15:25:49,212 -- Process Upload complete
debug - 2024-06-25 15:25:49,212 -- Upload result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 401', params={}, description='{\"detail\":\"Invalid token header. No credentials provided.\"}'), warnings=[], status_code=401, text='{\"detail\":\"Invalid token header. No credentials provided.\"}')"}
error - 2024-06-25 15:25:49,212 -- Upload failed: {"detail":"Invalid token header. No credentials provided."}
It is still considered successful but the coverage report/statuses are missing.
Acceptance criteria¶
- AC1: Coverage checks in the openQA-CI work
Suggestions¶
- Done Check if the issue is sporadic -> Not sporadic. Haven't seen a single successful upload recently.
- Done Check if we need to renew a token -> No, it still works on os-autoinst and we use the same token there. (The token secret in CircleCI is also correct judging by the last 4 letters being displayed in the UI.)
- Consider limit the way to reproduce by just trying to upload the report. No need to reconduct all tests with coverage for that
- Check whether the problem happens as well on https://github.com/openSUSE/open-build-service/pulls
- After a brief look it works there so it would make sense to see whether the CircleCI/Codecov setup on OBS is different. OBS is using CodeClimate primarily but they also still use codecov and stuff seems to still work on their side (see e.g. https://app.circleci.com/pipelines/github/openSUSE/open-build-service/19809/workflows/abbe028e-011e-44ab-9a1e-dd4fe55a9054/jobs/191497).
- Check https://github.com/codecov/feedback/issues (former https://community.codecov.com) or https://discuss.circleci.com (or just Google) for the problem
- After a brief look it doesn't seem to be a common issue. Maybe we can ask on those forums for help.
- Done Check whether the problem exists on os-autoinst (where we use GitHub actions instead of CircleCI) -> os-autoinst is fine, see https://github.com/os-autoinst/os-autoinst/pull/2509#issuecomment-2210522863 or https://github.com/os-autoinst/os-autoinst/actions/runs/9467154791/job/26080525367?pr=2501
Workaround¶
Look at the coverage report uploaded as CircleCI artifact and merge PRs manually if it looks good.
Updated by mkittler 4 months ago
If I download the binary that is also used within CircleCI and run it locally with the same params as on CircleCI (/hdd/downloads/codecov -v create-commit -t …) I don't run into the error we get within CircleCI. Maybe the token in CircleCI is not correct after all (not sure as it is redacted).
Updated by mkittler 4 months ago
Maybe it helps to specify the context in which the token is configured as documented on https://circleci.com/docs/env-vars. PR: https://github.com/os-autoinst/openQA/pull/5750
If that helps the token was never required before but now it is.
I also had a look on https://circleci.com/developer/orbs/orb/codecov/codecov?version=4.1.0 but couldn't find a mistake in our usage of the orb. The orb also hasn't changed much.
Note that OBS doesn't use the orb. They just download/run the uploader manually, see https://github.com/openSUSE/open-build-service/blob/d29bd63d406582221d69dd5938a19ee7534bd674/.circleci/conditional_config.yml#L377. They don't configure the token anywhere so I have no idea how the OBS setup works if the token is really required now (which might be the case as mentioned in the first paragraph).
Updated by openqa_review 4 months ago
- Due date set to 2024-07-23
Setting due date based on mean cycle time of SUSE QE Tools
Updated by mkittler 4 months ago
Only when trying to create my own post the forum suggested to look into https://discuss.circleci.com/t/unable-to-get-project-environment-variables-set-in-project/37433/4 first. So maybe we just have to follow https://circleci.com/docs/oss/#pass-secrets-to-builds-from-forked-pull-requests.
Updated by mkittler 4 months ago
- Status changed from In Progress to Feedback
This seems to work, see https://github.com/os-autoinst/openQA/pull/5750.
Updated by mkittler 4 months ago
Do you think we can improve the error reporting to not silently accept and then fail on empty/inaccessible tokens?
For this we would probably need to parse the output of codecov (the uploader binary). It unfortunately returns 0 in the error case (at least in this error case). I verified this locally. According to its --help it also has no flag to change that behavior. To parse the output we would probably need to invoke codecov directly (not using the Orb anymore). I think that's not worth it.
Updated by mkittler 4 months ago
I created https://github.com/os-autoinst/openQA/pull/5753 so despite the new setting the GitHub token won't be accessible from forked repos. For this I created a new context and restricted it to the tools team and our bot. I guess I'll temporarily remove the tools team and see whether the token is actually not present anymore on a PR from my fork.
If everything works as expected I'd consider this ticket resolved because improving the codecov upload further is probably not worth it.
Updated by mkittler 4 months ago
- Status changed from Feedback to Resolved
It didn't work because the user "Scheduled" (which is used by CircleCI to invoke the nightly job) is blocked by the context, see https://app.circleci.com/pipelines/github/os-autoinst/openQA/13977/workflows/a12d1b3b-6f54-4560-b081-bc0a5950e949.
So the blocking actually works but goes too far. I just somehow expected that the CircleCI user has some special role and it would work there regardless. There seems to be no way to allow the "Scheduled" user as this is only about our own teams we define in our GitHub orga. So I added "All members" back and instead restricted the context by branch. The relevant branch is "master". This seems to work now. I re-triggered the existing job (which means the associated user is still "Scheduled" and not me) and it works, see https://app.circleci.com/pipelines/github/os-autoinst/openQA/13977/workflows/372fd2ac-b5f7-4d45-9b5b-16b85c42bfda.
This shows that the context is effective so I'm confident it'll work when one of those CI jobs actually need to create a PR. So I'm resolving this issue for now.