action #69058
openqa-review fails to download job module detail step text info files since around 2020-07-08
0%
Description
Observation¶
https://gitlab.suse.de/openqa/openqa-review/-/pipelines shows that at "Jul 8, 2020 10:41am GMT+0200" everything still worked, the next run at "Jul 9, 2020 10:02am GMT+0200" failed with
++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.nd7hm8SHfZ
65 ........ERROR:openqa_review.openqa_review:Failed to process {'state': 'STILL_SOFT_FAILING', 'prev': {'href': '/tests/4391688'}, 'href': '/tests/4411953', 'failedmodules': []} with error Request to https://openqa.suse.de/tests/4411953/file/journalctl-43.txt was not successful, status code: 404. Skipping current result
…
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 479236 (char 479235)
Steps to reproduce¶
python openqa_review/openqa_review.py -vvvv -n -r -T --no-empty-sections --include-softfails -J https://openqa.suse.de/group_overview/290
Problem¶
Judging from the time I assume the issue was introduced within the following commits:
$ git log1 --no-merges bedf7a0de..484f4fb6d cafc8436b deps: Update Mojo-IOLoop-ReadWriteProcess 5bf9c4e40 Remove obsolete jobs table column 'backend' be0d11c51 mergify: Add additional automatic merge on special label 74fa29503 mergify: Only merge automatically when there are no pending reviews 119a31bae deps: Copy update-deps from os-autoinst c96bc9ceb Expand the search range in used/match when querying needles 5dd2d37d0 (okurz/enhance/systemd_dependencies) systemd: Prevent worker erroneously pulling in any local openQA webUI service 5916c0b83 Handle the case where a concurrent cache service Minion job failed gracefully 327dc0290 (Martchus/update-tidy) Update Perl::Tidy to 20200619 46240bfc8 (okurz/enhance/mergify) mergify: Also block merge on selected labels fd3452c8a (kalikiana/auth_oauth2_github) Add perl-Mojolicious-Plugin-OAuth2 dep 48b2b54ff Document how to configure OAuth 2.0 f9679bdc6 Capture output of Fake/OpenID cases and use tempdir 6112f2d06 Add OAuth2 auth method with GitHub provider b2c072829 Handle the case where the first cache service Minion job failed gracefully 69dbc0729 (perlpunk/updatedeps) deps: Make cpanfile targets configurable ee6482c8a (okurz/feature/rename_ci_setup) .circleci: Rename dependencies.txt to describe the actual use case 2dcd03800 (okurz/enhance/custom_clone) openqa-clone-custom-git-refspec: Enhance error message when called with wrong certificates 4151bf478 Simplify code for reading JSON files caabf1a5d Test OpenQA::CacheService::Client status functionality bfa192aff (ilausuch/sumarize_results_in_tab_2) Add a job summary into the detail page a48572c33 Extract renderTestSummary as a common helper to print the job summary c0372dbf6 Test OpenQA::CacheService::Client info functionality 767571a4e Test OpenQA::CacheService::Client enqueue functionality f810ec4dc Be more defensive and forward all cache service errors 3c5395167 Handle all cache service client errors consistently 5c4e893b2 Pass along cache service enqueue errors so they can be logged by the worker d8262f9b0 Dependency cron 200701 4fe2dc8b6 Copy update-deps script from os-autoinst 70c11bb5f Fix showing help for commands loaded via plugins faf24f340 (kalikiana/api_fixtures_glob) Narrow down fixtures needed by api tests 078f61d8f (kalikiana/t_fixtures_glob) Narrow down fixtures needed by toplevel tests
potential candidates for breakage as I consider:
- 4151bf478 Simplify code for reading JSON files
- d8262f9b0 Dependency cron 200701
also what I found that works is
curl 'https://openqa.suse.de/tests/4454659/modules/journalctl/steps/43' -H 'X-Requested-With: XMLHttpRequest'
but just
curl 'https://openqa.suse.de/tests/4454659/modules/journalctl/steps/43'
does not. Has this changed?
History
#1
Updated by mkittler 6 months ago
Updated by According to your message in Rocket Chat the missing text information is the problem. E.g. https://openqa.suse.de/tests/4454659/file/journalctl-43.txt is now 404 instead of returning the text from https://openqa.suse.de/tests/4454659#step/journalctl/43.
That has changed, indeed. The text is now incorporated into the JSON file when the job has finished to improve performance. So now the text is either still within the txt file or in the JSON file when the Minion job to finalize the openQA job has already run. This is not a bug because this is not part of the API. The JSON file can be found here: https://openqa.suse.de/tests/4454659/file/details-journalctl.json
I suggest to change the review script to download the JSON file from the mentioned URL and read the text_data from there. If it is missing you can still try to download the file reference via the text key because it might mean that the job hasn't finalized yet and the text is still present as a separate file.
Alternatively we could also provide an API for that.
Updated by #5
Updated by okurz 6 months ago
PR merged. Tagged the version with git tag -f 1.20.1 && git push -f origin 1.20.1 && git push -f origin HEAD:master for a nice package version in https://build.opensuse.org/package/show/home:okurz/python-openqa_review and called https://build.opensuse.org/package/view_file/devel:languages:python/python-openqa_review/_update.sh?expand=1 which created https://build.opensuse.org/request/show/822609 . https://download.opensuse.org/repositories/home:/okurz/openSUSE_Leap_15.1/noarch/ has the new packages. Tomorrow morning a cron job will auto-update packages on the machine lord.arch.suse.de and another cron job will trigger openqa-review to generate reports based on o3 and osd and publish them to https://w3.nue.suse.com/~okurz/openqa_opensuse_org_status.html and https://w3.nue.suse.com/~okurz/openqa_suse_de_status.html and others. These reports have short-links (without "okurz" in the name) as provided on https://progress.opensuse.org/projects/qa/wiki/Wiki#Test-results-overview . There is also https://gitlab.suse.de/openqa/openqa-review/pipelines that should eventually take over the job from lord.arch but is blocked by #52655 . The gitlab CI pipeline uses https://build.suse.de/package/show/home:okurz:container/openqa-review which was last built last midnight so does not have the new version yet. We should wait for this one to automatically install the new version of python-openqa_review as well and then monitor https://gitlab.suse.de/openqa/openqa-review/pipelines for no errors.
#6
Updated by okurz 6 months ago
- Status changed from In Progress to Feedback
https://w3.nue.suse.com/~okurz/openqa_sle15_functional_status.html looks fine again but we have error reports like in https://gitlab.suse.de/openqa/openqa-review/-/jobs/236665 :
Traceback (most recent call last):
File "/usr/bin/openqa-review", line 11, in <module>
load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')()
File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1295, in main
report = generate_report(args)
File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1229, in generate_report
return Report(browser, args, root_url, job_groups)
File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1180, in __init__
self.report[k] = self._one_report(v)
File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1191, in _one_report
return ProductReport(self.browser, job_group_url, self.root_url, self.args)
File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1003, in __init__
self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser)
File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 823, in __init__
self._search_for_bugrefs_for_softfailures(results)
File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 859, in _search_for_bugrefs_for_softfailures
v['bugref'] = self._get_bugref_for_softfailed_module(v, module_name)
File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 912, in _get_bugref_for_softfailed_module
details_json = json.loads(self.test_browser.get_soup(details_url).getText())
File "/usr/lib64/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 479236 (char 479235)
and we are still waiting for https://build.suse.de/package/show/home:okurz:container/openqa-review
With
podman run --rm -it registry.suse.de/home/okurz/container/images/openqa-review:latest /bin/sh -c 'rpm -qa | grep openqa_review'
one can check the version of openqa_review as available to the gitlab CI pipelines. Still showing "1.20.0-1.1" as of now.