Project

General

Profile

Actions
Copy link

action #156514

closed

Cron <root@openqa-service> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log failed / simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) / bugzilla_issue.py

Added by livdywan about 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
High
Assignee:
dheidler
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-07-18
Due date:
% Done:

0%

Estimated time:
Tags:
alert, reactive work, openqa-bugs

Description

Observation

Cron root@openqa-service (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log at Mon, 4 Mar 2024 00:20:40 +0000 (UTC):

Exception occured while fetching bsc#1158056
Traceback (most recent call last):
  File "/usr/bin/fetch_openqa_bugs", line 62, in <module>
    raise e
  File "/usr/bin/fetch_openqa_bugs", line 48, in <module>
    issue = issue_fetcher.get_issue(bugid)
  File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 88, in get_issue
    return self.prefix_table[prefix](self.conf, bugid)
  File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 24, in __init__
    self.fetch(conf)
  File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/bugzilla_issue.py", line 28, in fetch
    data = req.json()
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Suggestions

  • Investigate what is causing the JSON error - likely an error string was returned instead of JSON
  • Handle the exception in bugzilla_issue.py and print a meaningful error
  • Investigate why 3 seemingly identical errors are sent in separate emails around the same time
Actions
Copy link
 #1

Updated by livdywan about 2 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by mkittler about 2 months ago

  • Status changed from New to In Progress
  • Assignee set to mkittler
Actions
Copy link
 #3

Updated by mkittler about 2 months ago

  • Assignee changed from mkittler to dheidler

PR by @dheidler and me: https://github.com/os-autoinst/openqa_bugfetcher/pull/10

This will fix the error handling so it works as intended also when simplejson is used (instead of Python's normal JSON library). This should suppress non-critical errors like this one from showing up as alerts.

Actions
Copy link
 #4

Updated by dheidler about 2 months ago

  • Status changed from In Progress to Resolved
Actions
Copy link
 #5

Updated by livdywan about 2 months ago

  • Status changed from Resolved to Workable

mkittler wrote in #note-3:

PR by @dheidler and me: https://github.com/os-autoinst/openqa_bugfetcher/pull/10

This will fix the error handling so it works as intended also when simplejson is used (instead of Python's normal JSON library). This should suppress non-critical errors like this one from showing up as alerts.

Apparently this is not enough as-is and results in the following error on openqa-service in production:

Traceback (most recent call last):
  File "/usr/bin/fetch_openqa_bugs", line 58, in <module>
    except (json.decoder.JSONDecodeError, simplejson.errors.JSONDecodeError) as e:
NameError: name 'simplejson' is not defined
Actions
Copy link
 #6

Updated by mkittler about 2 months ago

Looks like we need to make this work in case simplejson is available and in the case when it is not. (And if it is available we need to make sure the type simplejson.errors.JSONDecodeError is actually imported.)

Actions
Copy link
 #7

Updated by dheidler about 2 months ago

  • Status changed from Workable to Resolved

https://github.com/os-autoinst/openqa_bugfetcher/commit/e232cb0b75e59e8d2aeb81cbbb75a10b5a934bce

Applied to openqa-service.
Simplified the bugfetcher rpm release process.
All that is needed to get a new rpm is to click on "trigger service run" at https://build.opensuse.org/package/show/devel:openQA/openqa_bugfetcher

Actions
Copy link
 #8

Updated by tinita about 2 months ago

Noe there seems to be an issue in the exception handling:

Date: Sat,  9 Mar 2024 00:01:29 +0000 (UTC)

Traceback (most recent call last):
  File "/usr/bin/fetch_openqa_bugs", line 49, in <module>
    issue = issue_fetcher.get_issue(bugid)
  File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 88, in get_issue
    return self.prefix_table[prefix](self.conf, bugid)
  File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 24, in __init__
    self.fetch(conf)
  File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/bugzilla_issue.py", line 28, in fetch
    data = req.json()
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/fetch_openqa_bugs", line 60, in <module>
    print("     -> ERROR: %s: JSON: %s…" % e, sys.last_value.doc[:50])
TypeError: not enough arguments for format string
Actions
Copy link
 #9

Updated by tinita about 2 months ago

  • Status changed from Resolved to Workable
Actions
Copy link
 #10

Updated by dheidler about 2 months ago

  • Status changed from Workable to Resolved

The args of that error str format had to be a tuple. Fixed and applied.

Actions
Copy link
 #11

Updated by livdywan about 2 months ago

dheidler wrote in #note-10:

The args of that error str format had to be a tuple. Fixed and applied.

I guess this is the fix: https://github.com/os-autoinst/openqa_bugfetcher/commit/c314739c7f0dbb9d0c21e99b34f38d8582c2f684

(In case anyone else got confused. Apparently there was no PR for this fix)

Actions
Copy link
 #12

Updated by livdywan about 1 month ago

  • Status changed from Resolved to Workable
  • Assignee deleted (dheidler)

I'm afraid it's still (or again) failing in much the same way:

    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/fetch_openqa_bugs", line 60, in <module>
    print("     -> ERROR: %s: JSON: %s…" % (e, sys.last_value.doc[:50]))
AttributeError: module 'sys' has no attribute 'last_value'

I think we should really have unit test coverage for this, since it's come back repeatedly with different errors in the same place. And probably check with multiple python environments (using tox or a matrix on GHA).

Actions
Copy link
 #13

Updated by dheidler about 1 month ago

  • Status changed from Workable to Resolved

The problem is that I can't really test that code path easily.

Should be fixed via https://github.com/os-autoinst/openqa_bugfetcher/commit/9213561fbf40a6dc9b8cd83be3c572fb43515950

Actions
Copy link
 #14

Updated by dheidler about 1 month ago

  • Assignee set to dheidler
Actions
Copy link

Also available in: Atom PDF