Project

General

Profile

Actions
Copy link

action #69058

closed

openqa-review fails to download job module detail step text info files since around 2020-07-08

Added by okurz over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
High
Assignee:
mkittler
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2020-07-16
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://gitlab.suse.de/openqa/openqa-review/-/pipelines shows that at "Jul 8, 2020 10:41am GMT+0200" everything still worked, the next run at "Jul 9, 2020 10:02am GMT+0200" failed with

++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.nd7hm8SHfZ
65 ........ERROR:openqa_review.openqa_review:Failed to process {'state': 'STILL_SOFT_FAILING', 'prev': {'href': '/tests/4391688'}, 'href': '/tests/4411953', 'failedmodules': []} with error Request to https://openqa.suse.de/tests/4411953/file/journalctl-43.txt was not successful, status code: 404. Skipping current result
…
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 479236 (char 479235)

Steps to reproduce

python openqa_review/openqa_review.py -vvvv -n -r -T --no-empty-sections --include-softfails -J https://openqa.suse.de/group_overview/290

Problem

Judging from the time I assume the issue was introduced within the following commits:

$ git log1 --no-merges bedf7a0de..484f4fb6d
cafc8436b deps: Update Mojo-IOLoop-ReadWriteProcess
5bf9c4e40 Remove obsolete jobs table column 'backend'
be0d11c51 mergify: Add additional automatic merge on special label
74fa29503 mergify: Only merge automatically when there are no pending reviews
119a31bae deps: Copy update-deps from os-autoinst
c96bc9ceb Expand the search range in used/match when querying needles
5dd2d37d0 (okurz/enhance/systemd_dependencies) systemd: Prevent worker erroneously pulling in any local openQA webUI service
5916c0b83 Handle the case where a concurrent cache service Minion job failed gracefully
327dc0290 (Martchus/update-tidy) Update Perl::Tidy to 20200619
46240bfc8 (okurz/enhance/mergify) mergify: Also block merge on selected labels
fd3452c8a (kalikiana/auth_oauth2_github) Add perl-Mojolicious-Plugin-OAuth2 dep
48b2b54ff Document how to configure OAuth 2.0
f9679bdc6 Capture output of Fake/OpenID cases and use tempdir
6112f2d06 Add OAuth2 auth method with GitHub provider
b2c072829 Handle the case where the first cache service Minion job failed gracefully
69dbc0729 (perlpunk/updatedeps) deps: Make cpanfile targets configurable
ee6482c8a (okurz/feature/rename_ci_setup) .circleci: Rename dependencies.txt to describe the actual use case
2dcd03800 (okurz/enhance/custom_clone) openqa-clone-custom-git-refspec: Enhance error message when called with wrong certificates
4151bf478 Simplify code for reading JSON files
caabf1a5d Test OpenQA::CacheService::Client status functionality
bfa192aff (ilausuch/sumarize_results_in_tab_2) Add a job summary into the detail page
a48572c33 Extract renderTestSummary as a common helper to print the job summary
c0372dbf6 Test OpenQA::CacheService::Client info functionality
767571a4e Test OpenQA::CacheService::Client enqueue functionality
f810ec4dc Be more defensive and forward all cache service errors
3c5395167 Handle all cache service client errors consistently
5c4e893b2 Pass along cache service enqueue errors so they can be logged by the worker
d8262f9b0 Dependency cron 200701
4fe2dc8b6 Copy update-deps script from os-autoinst
70c11bb5f Fix showing help for commands loaded via plugins
faf24f340 (kalikiana/api_fixtures_glob) Narrow down fixtures needed by api tests
078f61d8f (kalikiana/t_fixtures_glob) Narrow down fixtures needed by toplevel tests

potential candidates for breakage as I consider:

  • 4151bf478 Simplify code for reading JSON files
  • d8262f9b0 Dependency cron 200701

also what I found that works is

curl 'https://openqa.suse.de/tests/4454659/modules/journalctl/steps/43' -H 'X-Requested-With: XMLHttpRequest'

but just

curl 'https://openqa.suse.de/tests/4454659/modules/journalctl/steps/43'

does not. Has this changed?

Actions
Copy link
 #1

Updated by mkittler over 4 years ago

According to your message in Rocket Chat the missing text information is the problem. E.g. https://openqa.suse.de/tests/4454659/file/journalctl-43.txt is now 404 instead of returning the text from https://openqa.suse.de/tests/4454659#step/journalctl/43.

That has changed, indeed. The text is now incorporated into the JSON file when the job has finished to improve performance. So now the text is either still within the txt file or in the JSON file when the Minion job to finalize the openQA job has already run. This is not a bug because this is not part of the API. The JSON file can be found here: https://openqa.suse.de/tests/4454659/file/details-journalctl.json

I suggest to change the review script to download the JSON file from the mentioned URL and read the text_data from there. If it is missing you can still try to download the file reference via the text key because it might mean that the job hasn't finalized yet and the text is still present as a separate file.

Alternatively we could also provide an API for that.

Actions
Copy link
 #2

Updated by okurz over 4 years ago

  • Status changed from New to Workable
Actions
Copy link
 #3

Updated by mkittler over 4 years ago

  • Assignee set to mkittler
Actions
Copy link
 #4

Updated by mkittler over 4 years ago

  • Status changed from Workable to In Progress
Actions
Copy link
 #5

Updated by okurz over 4 years ago

PR merged. Tagged the version with git tag -f 1.20.1 && git push -f origin 1.20.1 && git push -f origin HEAD:master for a nice package version in https://build.opensuse.org/package/show/home:okurz/python-openqa_review and called https://build.opensuse.org/package/view_file/devel:languages:python/python-openqa_review/_update.sh?expand=1 which created https://build.opensuse.org/request/show/822609 . https://download.opensuse.org/repositories/home:/okurz/openSUSE_Leap_15.1/noarch/ has the new packages. Tomorrow morning a cron job will auto-update packages on the machine lord.arch.suse.de and another cron job will trigger openqa-review to generate reports based on o3 and osd and publish them to https://w3.nue.suse.com/~okurz/openqa_opensuse_org_status.html and https://w3.nue.suse.com/~okurz/openqa_suse_de_status.html and others. These reports have short-links (without "okurz" in the name) as provided on https://progress.opensuse.org/projects/qa/wiki/Wiki#Test-results-overview . There is also https://gitlab.suse.de/openqa/openqa-review/pipelines that should eventually take over the job from lord.arch but is blocked by #52655 . The gitlab CI pipeline uses https://build.suse.de/package/show/home:okurz:container/openqa-review which was last built last midnight so does not have the new version yet. We should wait for this one to automatically install the new version of python-openqa_review as well and then monitor https://gitlab.suse.de/openqa/openqa-review/pipelines for no errors.

Actions
Copy link
 #6

Updated by okurz over 4 years ago

  • Status changed from In Progress to Feedback

https://w3.nue.suse.com/~okurz/openqa_sle15_functional_status.html looks fine again but we have error reports like in https://gitlab.suse.de/openqa/openqa-review/-/jobs/236665 :

Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 11, in <module>
    load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')()
  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1295, in main
    report = generate_report(args)
  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1229, in generate_report
    return Report(browser, args, root_url, job_groups)
  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1180, in __init__
    self.report[k] = self._one_report(v)
  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1191, in _one_report
    return ProductReport(self.browser, job_group_url, self.root_url, self.args)
  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1003, in __init__
    self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser)
  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 823, in __init__
    self._search_for_bugrefs_for_softfailures(results)
  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 859, in _search_for_bugrefs_for_softfailures
    v['bugref'] = self._get_bugref_for_softfailed_module(v, module_name)
  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 912, in _get_bugref_for_softfailed_module
    details_json = json.loads(self.test_browser.get_soup(details_url).getText())
  File "/usr/lib64/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 479236 (char 479235)

and we are still waiting for https://build.suse.de/package/show/home:okurz:container/openqa-review

With

podman run --rm -it registry.suse.de/home/okurz/container/images/openqa-review:latest /bin/sh -c 'rpm -qa | grep openqa_review'

one can check the version of openqa_review as available to the gitlab CI pipelines. Still showing "1.20.0-1.1" as of now.

Actions
Copy link
 #7

Updated by okurz over 4 years ago

  • Status changed from Feedback to Resolved

well, ok. Let's handle that like another problem. I will submit a fix soon.

Actions
Copy link

Also available in: Atom PDF