QA » openQA Project

Observation¶

https://gitlab.suse.de/openqa/openqa-review/-/pipelines shows that at "Jul 8, 2020 10:41am GMT+0200" everything still worked, the next run at "Jul 9, 2020 10:02am GMT+0200" failed with

++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.nd7hm8SHfZ
65 ........ERROR:openqa_review.openqa_review:Failed to process {'state': 'STILL_SOFT_FAILING', 'prev': {'href': '/tests/4391688'}, 'href': '/tests/4411953', 'failedmodules': []} with error Request to https://openqa.suse.de/tests/4411953/file/journalctl-43.txt was not successful, status code: 404. Skipping current result
…
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 479236 (char 479235)

Steps to reproduce¶

python openqa_review/openqa_review.py -vvvv -n -r -T --no-empty-sections --include-softfails -J https://openqa.suse.de/group_overview/290

Problem¶

Judging from the time I assume the issue was introduced within the following commits:

$ git log1 --no-merges bedf7a0de..484f4fb6d
cafc8436b deps: Update Mojo-IOLoop-ReadWriteProcess
5bf9c4e40 Remove obsolete jobs table column 'backend'
be0d11c51 mergify: Add additional automatic merge on special label
74fa29503 mergify: Only merge automatically when there are no pending reviews
119a31bae deps: Copy update-deps from os-autoinst
c96bc9ceb Expand the search range in used/match when querying needles
5dd2d37d0 (okurz/enhance/systemd_dependencies) systemd: Prevent worker erroneously pulling in any local openQA webUI service
5916c0b83 Handle the case where a concurrent cache service Minion job failed gracefully
327dc0290 (Martchus/update-tidy) Update Perl::Tidy to 20200619
46240bfc8 (okurz/enhance/mergify) mergify: Also block merge on selected labels
fd3452c8a (kalikiana/auth_oauth2_github) Add perl-Mojolicious-Plugin-OAuth2 dep
48b2b54ff Document how to configure OAuth 2.0
f9679bdc6 Capture output of Fake/OpenID cases and use tempdir
6112f2d06 Add OAuth2 auth method with GitHub provider
b2c072829 Handle the case where the first cache service Minion job failed gracefully
69dbc0729 (perlpunk/updatedeps) deps: Make cpanfile targets configurable
ee6482c8a (okurz/feature/rename_ci_setup) .circleci: Rename dependencies.txt to describe the actual use case
2dcd03800 (okurz/enhance/custom_clone) openqa-clone-custom-git-refspec: Enhance error message when called with wrong certificates
4151bf478 Simplify code for reading JSON files
caabf1a5d Test OpenQA::CacheService::Client status functionality
bfa192aff (ilausuch/sumarize_results_in_tab_2) Add a job summary into the detail page
a48572c33 Extract renderTestSummary as a common helper to print the job summary
c0372dbf6 Test OpenQA::CacheService::Client info functionality
767571a4e Test OpenQA::CacheService::Client enqueue functionality
f810ec4dc Be more defensive and forward all cache service errors
3c5395167 Handle all cache service client errors consistently
5c4e893b2 Pass along cache service enqueue errors so they can be logged by the worker
d8262f9b0 Dependency cron 200701
4fe2dc8b6 Copy update-deps script from os-autoinst
70c11bb5f Fix showing help for commands loaded via plugins
faf24f340 (kalikiana/api_fixtures_glob) Narrow down fixtures needed by api tests
078f61d8f (kalikiana/t_fixtures_glob) Narrow down fixtures needed by toplevel tests

potential candidates for breakage as I consider:

• 4151bf478 Simplify code for reading JSON files
• d8262f9b0 Dependency cron 200701

also what I found that works is

curl 'https://openqa.suse.de/tests/4454659/modules/journalctl/steps/43' -H 'X-Requested-With: XMLHttpRequest'

but just

curl 'https://openqa.suse.de/tests/4454659/modules/journalctl/steps/43' 

does not. Has this changed?