action #120525
closedopenQA Infrastructure - coordination #120522: [epic] Support upgrade SUSE bugzilla instance
Ensure our usual bugzilla integrated tooling works with the upgraded test instance size:M
0%
Description
Motivation¶
SUSE upgrades the SUSE bugzilla instance which can bring some changes to how we interact with it with our tooling, e.g. openqa-review, etc.
Acceptance criteria¶
- AC1: bug reading and bug comment writing is ensured to work with openqa-review against the upgraded test instance
- AC2: other tools have been checked
Suggestions¶
- Follow discussions e.g. in https://suse.slack.com/archives/C02CANHLANP/p1668421082742279 and https://suse.slack.com/archives/C02R5HRKD1C/p1668177301594739 about JSON-RPC
- Find out details about the test instance, URL, how to connect, how to authenticate
- Conduct preliminary tests, e.g. simple curl command lines to retrieve bug data and write comments on bugs -> already done, see comments
- Crosscheck if openqa-review can work with that upgraded test instance including sending reminder comments on bugs
- Check if mtui accesses bugzilla, e.g. just call git grep
- Optional: Within github.com/openSUSE/ look for "bugzilla" or "XMLRPC" or "JSONRPC", see https://github.com/search?q=org%3Aos-autoinst+jsonrpc.cgi&type=code
- Check that the least intrusive approach i.e. using the new authentication headers work via the "next" test instance, see also #note-6
Out of scope¶
Updated by okurz about 2 years ago
- Priority changed from Normal to High
We expect the upgraded test instance to go live within the next four weeks so we should test within this timeframe.
Updated by okurz about 2 years ago
- Assignee set to okurz
According to https://suse.slack.com/archives/C02R5HRKD1C/p1668508452624309?thread_ts=1668177301.594739&cid=C02R5HRKD1C
(Johannes Van Wijk) I don't think we will remove [JSON-RPC], unless we upgrade to a newer version in the future, but I guess that will not be in the next 2 years...
(Vladimir Siman) exactly; it's already deprecated and not being developed; and will be removed at some point in some future version; but it's not gonna be removed by us and with this upgrade project
I asked in https://suse.slack.com/archives/C02R5HRKD1C/p1668510117873549 about URL and credentials
Updated by okurz about 2 years ago
- Status changed from Feedback to New
- Assignee deleted (
okurz) - Priority changed from High to Normal
(Michal Pavúk) The URL is https://bugzilla-next.suse.de/ (VPN required either corp, or engineering), I'm not sure if there are plans for publicly available "next" version. For interactive login you can use your univention idp credentials (the ones used for current bugzilla/IBS/confluence).
I'm not sure about API access, if I understood that correctly you can issue API tokens via the web interface.
@Vladimir Siman will know more.
Updated by jstehlik about 2 years ago
- Priority changed from Normal to High
Info on how to get API key: log in using your IDP credentials, then in your account preferences in Bugzilla create an API key.
https://confluence.suse.com/display/LITC/Bugzilla+-+API+access+howto
For the live phase opanqa needs to get its own key and thus possibly also an IDP account.
Updated by okurz about 2 years ago
I just tried it out. I logged in over the browser to https://bugzilla-next.suse.de/ going over the IDP authentication provider, same as we use for IBS. Then over the menues "Preferences", "API KEYS" I generated an API key and called
$ curl -s -H 'Bugzilla_login: okurz@suse.com' -H 'Bugzilla_api_key: XXX' 'https://bugzilla-next.suse.de/jsonrpc.cgi?method=Bug.get¶ms=%5B%7B%22ids%22%3A+%5B1204372%5D%7D%5D'
and I got
{"error":null,"result":{"faults":[],"bugs":[{"id":1204372,"is_cc_accessible":true,"cf_foundby":"---","product":"openSUSE Distribution","see_also":[],"creator":"caochenzi@163.com","priority":"P5 - None","creation_time":"2022-10-26T11:22:39Z","alias":[],"severity":"Major","flags":[],"blocks":[],"cf_biz_priority":"","component":"Basesystem","resolution":"","is_creator_accessible":true,"qa_contact":"qa-bugs@suse.de","assigned_to":"screening-team-bugs@suse.de","classification":"openSUSE","whiteboard":"","last_change_time":"2022-10-26T11:22:39Z","cf_blocker":"---","cf_marketing_qa_status":"---","dupe_of":null,"summary":"test filing bugs","depends_on":[],"op_sys":"Other","is_open":true,"cc_detail":[],"cc":[],"creator_detail":{"id":91750,"email":"caochenzi@163.com","real_name":"chenzi cao","name":"caochenzi@163.com"},"assigned_to_detail":{"id":87664,"real_name":"E-mail List","email":"screening-team-bugs@suse.de","name":"screening-team-bugs@suse.de"},"version":"Leap 15.3","groups":[],"status":"NEW","qa_contact_detail":{"id":7641,"name":"qa-bugs@suse.de","real_name":"E-mail List","email":"qa-bugs@suse.de"},"url":"","cf_nts_priority":"","platform":"x86-64","is_confirmed":true,"keywords":[],"cf_it_deployment":"---","deadline":null,"target_milestone":"---"}]},"id":"https://bugzilla-next.suse.de/"}
so everything looks good.
For another bug which I could see in the webUI but on the CLI using my API key that failed:
{"error":{"message":"You are not authorized to access bug #1204384. To see this bug, you must first log in to an account with the appropriate permissions.","code":102},"result":null,"id":"https://bugzilla-next.suse.de/"}
Within our code e.g. in github.com/os-autoinst/openqa_review we are passing the "auth" object which includes username and password. Likely we just need to change that part, e.g. see openqa_review/browser.py +139
Updated by livdywan about 2 years ago
- Subject changed from Ensure our usual bugzilla integrated tooling works with the upgraded test instance to Ensure our usual bugzilla integrated tooling works with the upgraded test instance size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by okurz about 2 years ago
- Related to action #120276: Prepare openqa_bugfetcher for upcoming bugzilla update added
Updated by jstehlik almost 2 years ago
Thanks Oliver for performing integration smoke test. Sign-off has been planned for end of this week and I confirmed with Cris, that during next week we should get ready for the new login method. This week my goal is to obtain key for openQA
Updated by okurz almost 2 years ago
As necessary there is a new user account which we can use, see https://gitlab.suse.de/openqa/password/-/merge_requests/3 for credentials
Updated by robert.richardson almost 2 years ago
- Status changed from Workable to Blocked
Blocked due to 502 Bad Gateway
on bugzilla test instance
Updated by robert.richardson almost 2 years ago
- Status changed from Blocked to In Progress
- Assignee set to robert.richardson
https://bugzilla-next.suse.de/ is back online
Updated by robert.richardson almost 2 years ago
- Status changed from In Progress to Workable
@okurz i wasnt able to log in on the new test instance using the provided IDP credentials, tried "openqa-admins@suse.de" and "openqa-admins" as username plus the IPD pw from your link.
Since i also ran into problems trying to run tests and i'm going to be in school for two weeks i'm now marking this as workable again.
Updated by robert.richardson almost 2 years ago
- Assignee deleted (
robert.richardson)
Updated by osukup almost 2 years ago
MTUI has connection to bugzilla using XML-RPC but after removing testopia support from MTUI it now don't use this code
Updated by osukup almost 2 years ago
btw -> https://confluence.suse.com/display/LITC/Bugzilla+-+API+access+howto is strongly recommending REST API as XML-RPC is deprecated and JSON-RPC newer got over experimental support
Updated by livdywan almost 2 years ago
osukup wrote:
btw -> https://confluence.suse.com/display/LITC/Bugzilla+-+API+access+howto is strongly recommending REST API as XML-RPC is deprecated and JSON-RPC newer got over experimental support
I woudn't worry about the experimental status after having used it for so long 😉 The issue with using the new API is that we can't deploy it on the current version, so as a team we decided against doing that until the production instance provides it.
Updated by osukup almost 2 years ago
cdywan wrote:
osukup wrote:
btw -> https://confluence.suse.com/display/LITC/Bugzilla+-+API+access+howto is strongly recommending REST API as XML-RPC is deprecated and JSON-RPC newer got over experimental support
I woudn't worry about the experimental status after having used it for so long 😉 The issue with using the new API is that we can't deploy it on the current version, so as a team we decided against doing that until the production instance provides it.
--> used it for long , doesn't mean it cannot disappear --> co move to stable and preferred API isn't bad idea
Updated by osukup almost 2 years ago
>> curl -s -H 'Bugzilla_login: osukup@suse.com' -H 'Bugzilla_api_key: TOKEN' 'https://bugzilla-next.suse.de/jsonrpc.cgi?method=Bug.get¶ms=%5B%7B%22ids%22%3A+%5B1204372%5D%7D%5D'
{"result":null,"error":{"code":101,"message":"Bug #1204372 does not exist."},"id":"https://bugzilla-next.suse.de/"}
which looks like api works but returns invalid data ? after looking to instance it seems empty ...
- there is interesting part ... api examples in confluence have login and api key in params, not header ...?
Updated by osukup almost 2 years ago
hmm, funny thing -> example in confluence isn't valid and returns:
{'error': {'code': 32000, 'message': "When using JSON-RPC over GET, you must specify a 'method' parameter. See the documentation at https://bugzilla.readthedocs.org/en/5.0/api/Bugzilla/WebService/Server/JSONRPC.html"}, 'result': None, 'id': 'https://bugzilla-next.suse.de/'}
which is even more funny because mentioned documentation returns non-existing page .. in related bugzilla docu is mentioned XMLRPC and JSONRPC exists but links to pages with docu are invalid...
Updated by livdywan almost 2 years ago
- Description updated (diff)
osukup wrote:
{'error': {'code': 32000, 'message': "When using JSON-RPC over GET, you must specify a 'method' parameter. See the documentation at https://bugzilla.readthedocs.org/en/5.0/api/Bugzilla/WebService/Server/JSONRPC.html"}, 'result': None, 'id': 'https://bugzilla-next.suse.de/'}
curl: (7) Failed to connect to bugzilla-next.suse.de port 443 after 14 ms: Connection refused
For me it's simply not responding again. Mentioned this on ENGINFRA-1457.
Updated by livdywan almost 2 years ago
cdywan wrote:
For me it's simply not responding again. Mentioned this on ENGINFRA-1457.
Still waiting on this.
Updated by osukup almost 2 years ago
- Status changed from Blocked to In Progress
I'm working on change of openQA_review to be able use both apis ( if is api_key in config present use rest else json-rpc )
Updated by livdywan almost 2 years ago
osukup wrote:
I'm working on change of openQA_review to be able use both apis ( if is api_key in config present use rest else json-rpc )
How was this unblocked now? And why does it sound like you're pursuing a very different plan now than we discussed within the team? Some reasoning would be appreciated.
Updated by openqa_review almost 2 years ago
- Due date set to 2023-01-25
Setting due date based on mean cycle time of SUSE QE Tools
Updated by osukup almost 2 years ago
- Status changed from In Progress to Blocked
https://suse.slack.com/archives/C02AJ1E568M/p1673437437406349?thread_ts=1673437367.328629&cid=C02AJ1E568M -> @okurz rejecting move to REST api ... so blocked until JSON-RPC is working on staging instance with token
Updated by livdywan almost 2 years ago
- Due date deleted (
2023-01-25)
This is blocked and we don't have any current timeline so the due date is not helpful
Updated by livdywan almost 2 years ago
- Due date set to 2023-02-03
- Status changed from Blocked to Feedback
This isn't blocking on a ticket, but we need to confirm that we have an instance to test against.
Updated by livdywan almost 2 years ago
- Due date changed from 2023-02-03 to 2023-02-10
Bumping due date due to hackweek.
Updated by okurz almost 2 years ago
- Due date changed from 2023-02-10 to 2023-02-24
Discussed within the weekly unblock. https://bugzilla-next.suse.de/ is currently not available, showing Error 500.
Updated by okurz almost 2 years ago
@cdywan @osukup https://bugzilla-next.suse.de/ is reachable right now. I recommend you crosscheck our tooling against that instance with as little effort as possible.
Updated by okurz almost 2 years ago
- Status changed from Feedback to Workable
- Priority changed from High to Urgent
Updated by livdywan almost 2 years ago
- Status changed from Workable to In Progress
Ack. Checking right now
Updated by livdywan almost 2 years ago
For reference I'm testing with this snippet. It seems neither passing the token via headers nor params in the JSON or in the URL work... the docs say both should, and even just "token" should suffice but I'm always getting 102 for the private bug here:
for BUG in 1204372 1190670 1203578; do # non-existent public private
echo "https://bugzilla-next.suse.de/show_bug.cgi?id=$BUG"
curl -s -H "Bugzilla_login: $BZ_EMAIL" -H "Bugzilla_token: $BZ_API_KEY" "https://bugzilla-next.suse.de/jsonrpc.cgi?method=Bug.get¶ms=%5B%7B%22ids%22%3A+%5B$BUG%5D%7D%5D"
echo .
curl -s "https%3A%2F%2Fbugzilla-next.suse.de%2Fjsonrpc.cgi%3Fmethod%3DBug.get%26params%3D%5B%7B%22token%22%3A%22$BZ_API_KEY%22%2C%22ids%22%3A%2B%5B$BUG%5D%7D%5D"
echo .
done
I'm also checking with Martin to confirm how the token should be passed here since it feels like the syntax may just be wrong? I expect the error won't tell me more than whether the JSON's valid.
Updated by livdywan almost 2 years ago
Going ahead with a draft PR anyway to ensure things are easily testable, and maybe that also helps rule out mistakes made because of manual construction of URL's: https://github.com/os-autoinst/openqa_review/pull/219
Not sure how to fix the unit tests yet.
Updated by livdywan almost 2 years ago
okurz wrote:
- Check if mtui accesses bugzilla, e.g. just call git grep
With relevant code having been recently dropped I guess there's nothing to do for mtui?
Updated by livdywan almost 2 years ago
cdywan wrote:
Going ahead with a draft PR anyway to ensure things are easily testable, and maybe that also helps rule out mistakes made because of manual construction of URL's: https://github.com/os-autoinst/openqa_review/pull/219
Note that testing against the staging instance has one drawback. The database may not match. I solved this by locally injecting different bug numbers, i.e. something like this:
- self.bugid = int(bugid_match.group(2)) if bugid_match else None
+ self.bugid = 1203578 # 1190670
And both public and private bugs seem to work and show up in reports e.g. SKIP_SEND_MAIL=1 openqa-review -j ALP --query-issue-status
(I'm using pip install -e .
to run my local branch)
Updated by livdywan over 1 year ago
- Assignee changed from osukup to livdywan
Maybe I should take the ticket at this point since I'm driving the changes to auto_review.
What will need to be done eventually is to add an API key to https://gitlab.suse.de/openqa/openqa-review but this isn't possible until the new version is deployed in production.
Updated by livdywan over 1 year ago
- Status changed from In Progress to Feedback
Merged. Ideally nothing really changes and reports continue to be generated.
Updated by livdywan over 1 year ago
- Status changed from Feedback to Blocked
This is blocking on the actual upgrade now, see Slack.
Updated by livdywan over 1 year ago
- Due date deleted (
2023-02-24)
This shouldn't have a due date, since it's blocked
Updated by osukup over 1 year ago
- Related to action #127826: [tools] openqa-review CI pipelines failing added
Updated by osukup over 1 year ago
- Status changed from Blocked to Resolved
new bugzilla was deployed - and Gitlab CI fixed