Project

General

Profile

Actions

action #120525

closed

openQA Infrastructure - coordination #120522: [epic] Support upgrade SUSE bugzilla instance

Ensure our usual bugzilla integrated tooling works with the upgraded test instance size:M

Added by okurz about 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2022-11-15
Due date:
% Done:

0%

Estimated time:

Description

Motivation

SUSE upgrades the SUSE bugzilla instance which can bring some changes to how we interact with it with our tooling, e.g. openqa-review, etc.

Acceptance criteria

  • AC1: bug reading and bug comment writing is ensured to work with openqa-review against the upgraded test instance
  • AC2: other tools have been checked

Suggestions

Out of scope


Related issues 2 (0 open2 closed)

Related to openQA Infrastructure - action #120276: Prepare openqa_bugfetcher for upcoming bugzilla updateResolveddheidler2022-11-10

Actions
Related to openQA Project - action #127826: [tools] openqa-review CI pipelines failing Rejectedlivdywan2023-04-18

Actions
Actions #1

Updated by okurz about 2 years ago

  • Priority changed from Normal to High

We expect the upgraded test instance to go live within the next four weeks so we should test within this timeframe.

Actions #2

Updated by okurz about 2 years ago

  • Assignee set to okurz

According to https://suse.slack.com/archives/C02R5HRKD1C/p1668508452624309?thread_ts=1668177301.594739&cid=C02R5HRKD1C

(Johannes Van Wijk) I don't think we will remove [JSON-RPC], unless we upgrade to a newer version in the future, but I guess that will not be in the next 2 years...
(Vladimir Siman) exactly; it's already deprecated and not being developed; and will be removed at some point in some future version; but it's not gonna be removed by us and with this upgrade project

I asked in https://suse.slack.com/archives/C02R5HRKD1C/p1668510117873549 about URL and credentials

Actions #3

Updated by okurz about 2 years ago

  • Status changed from New to Feedback
Actions #4

Updated by okurz about 2 years ago

  • Status changed from Feedback to New
  • Assignee deleted (okurz)
  • Priority changed from High to Normal

(Michal Pavúk) The URL is https://bugzilla-next.suse.de/ (VPN required either corp, or engineering), I'm not sure if there are plans for publicly available "next" version. For interactive login you can use your univention idp credentials (the ones used for current bugzilla/IBS/confluence).
I'm not sure about API access, if I understood that correctly you can issue API tokens via the web interface.
@Vladimir Siman will know more.

Actions #5

Updated by jstehlik about 2 years ago

  • Priority changed from Normal to High

Info on how to get API key: log in using your IDP credentials, then in your account preferences in Bugzilla create an API key.
https://confluence.suse.com/display/LITC/Bugzilla+-+API+access+howto
For the live phase opanqa needs to get its own key and thus possibly also an IDP account.

Actions #6

Updated by okurz about 2 years ago

I just tried it out. I logged in over the browser to https://bugzilla-next.suse.de/ going over the IDP authentication provider, same as we use for IBS. Then over the menues "Preferences", "API KEYS" I generated an API key and called

$ curl -s -H 'Bugzilla_login: okurz@suse.com' -H 'Bugzilla_api_key: XXX' 'https://bugzilla-next.suse.de/jsonrpc.cgi?method=Bug.get&params=%5B%7B%22ids%22%3A+%5B1204372%5D%7D%5D'

and I got

{"error":null,"result":{"faults":[],"bugs":[{"id":1204372,"is_cc_accessible":true,"cf_foundby":"---","product":"openSUSE Distribution","see_also":[],"creator":"caochenzi@163.com","priority":"P5 - None","creation_time":"2022-10-26T11:22:39Z","alias":[],"severity":"Major","flags":[],"blocks":[],"cf_biz_priority":"","component":"Basesystem","resolution":"","is_creator_accessible":true,"qa_contact":"qa-bugs@suse.de","assigned_to":"screening-team-bugs@suse.de","classification":"openSUSE","whiteboard":"","last_change_time":"2022-10-26T11:22:39Z","cf_blocker":"---","cf_marketing_qa_status":"---","dupe_of":null,"summary":"test filing bugs","depends_on":[],"op_sys":"Other","is_open":true,"cc_detail":[],"cc":[],"creator_detail":{"id":91750,"email":"caochenzi@163.com","real_name":"chenzi cao","name":"caochenzi@163.com"},"assigned_to_detail":{"id":87664,"real_name":"E-mail List","email":"screening-team-bugs@suse.de","name":"screening-team-bugs@suse.de"},"version":"Leap 15.3","groups":[],"status":"NEW","qa_contact_detail":{"id":7641,"name":"qa-bugs@suse.de","real_name":"E-mail List","email":"qa-bugs@suse.de"},"url":"","cf_nts_priority":"","platform":"x86-64","is_confirmed":true,"keywords":[],"cf_it_deployment":"---","deadline":null,"target_milestone":"---"}]},"id":"https://bugzilla-next.suse.de/"}

so everything looks good.

For another bug which I could see in the webUI but on the CLI using my API key that failed:

{"error":{"message":"You are not authorized to access bug #1204384. To see this bug, you must first log in to an account with the appropriate permissions.","code":102},"result":null,"id":"https://bugzilla-next.suse.de/"}

Within our code e.g. in github.com/os-autoinst/openqa_review we are passing the "auth" object which includes username and password. Likely we just need to change that part, e.g. see openqa_review/browser.py +139

Actions #7

Updated by livdywan about 2 years ago

  • Subject changed from Ensure our usual bugzilla integrated tooling works with the upgraded test instance to Ensure our usual bugzilla integrated tooling works with the upgraded test instance size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #8

Updated by okurz about 2 years ago

  • Related to action #120276: Prepare openqa_bugfetcher for upcoming bugzilla update added
Actions #9

Updated by jstehlik almost 2 years ago

Thanks Oliver for performing integration smoke test. Sign-off has been planned for end of this week and I confirmed with Cris, that during next week we should get ready for the new login method. This week my goal is to obtain key for openQA

Actions #10

Updated by okurz almost 2 years ago

As necessary there is a new user account which we can use, see https://gitlab.suse.de/openqa/password/-/merge_requests/3 for credentials

Actions #11

Updated by robert.richardson almost 2 years ago

  • Status changed from Workable to Blocked
Actions #12

Updated by robert.richardson almost 2 years ago

  • Status changed from Blocked to In Progress
  • Assignee set to robert.richardson
Actions #13

Updated by robert.richardson almost 2 years ago

  • Status changed from In Progress to Workable

@okurz i wasnt able to log in on the new test instance using the provided IDP credentials, tried "openqa-admins@suse.de" and "openqa-admins" as username plus the IPD pw from your link.
Since i also ran into problems trying to run tests and i'm going to be in school for two weeks i'm now marking this as workable again.

Actions #14

Updated by robert.richardson almost 2 years ago

  • Assignee deleted (robert.richardson)
Actions #15

Updated by osukup almost 2 years ago

MTUI has connection to bugzilla using XML-RPC but after removing testopia support from MTUI it now don't use this code

Actions #16

Updated by osukup almost 2 years ago

btw -> https://confluence.suse.com/display/LITC/Bugzilla+-+API+access+howto is strongly recommending REST API as XML-RPC is deprecated and JSON-RPC newer got over experimental support

Actions #17

Updated by livdywan almost 2 years ago

osukup wrote:

btw -> https://confluence.suse.com/display/LITC/Bugzilla+-+API+access+howto is strongly recommending REST API as XML-RPC is deprecated and JSON-RPC newer got over experimental support

I woudn't worry about the experimental status after having used it for so long 😉 The issue with using the new API is that we can't deploy it on the current version, so as a team we decided against doing that until the production instance provides it.

Actions #18

Updated by osukup almost 2 years ago

cdywan wrote:

osukup wrote:

btw -> https://confluence.suse.com/display/LITC/Bugzilla+-+API+access+howto is strongly recommending REST API as XML-RPC is deprecated and JSON-RPC newer got over experimental support

I woudn't worry about the experimental status after having used it for so long 😉 The issue with using the new API is that we can't deploy it on the current version, so as a team we decided against doing that until the production instance provides it.

--> used it for long , doesn't mean it cannot disappear --> co move to stable and preferred API isn't bad idea

Actions #19

Updated by osukup almost 2 years ago

  • Assignee set to osukup
Actions #20

Updated by osukup almost 2 years ago

>> curl -s -H 'Bugzilla_login: osukup@suse.com' -H 'Bugzilla_api_key: TOKEN' 'https://bugzilla-next.suse.de/jsonrpc.cgi?method=Bug.get&params=%5B%7B%22ids%22%3A+%5B1204372%5D%7D%5D'
{"result":null,"error":{"code":101,"message":"Bug #1204372 does not exist."},"id":"https://bugzilla-next.suse.de/"}

which looks like api works but returns invalid data ? after looking to instance it seems empty ...

  • there is interesting part ... api examples in confluence have login and api key in params, not header ...?
Actions #21

Updated by osukup almost 2 years ago

hmm, funny thing -> example in confluence isn't valid and returns:

{'error': {'code': 32000, 'message': "When using JSON-RPC over GET, you must specify a 'method' parameter. See the documentation at https://bugzilla.readthedocs.org/en/5.0/api/Bugzilla/WebService/Server/JSONRPC.html"}, 'result': None, 'id': 'https://bugzilla-next.suse.de/'}

which is even more funny because mentioned documentation returns non-existing page .. in related bugzilla docu is mentioned XMLRPC and JSONRPC exists but links to pages with docu are invalid...

Actions #22

Updated by livdywan almost 2 years ago

  • Description updated (diff)

osukup wrote:

{'error': {'code': 32000, 'message': "When using JSON-RPC over GET, you must specify a 'method' parameter. See the documentation at https://bugzilla.readthedocs.org/en/5.0/api/Bugzilla/WebService/Server/JSONRPC.html"}, 'result': None, 'id': 'https://bugzilla-next.suse.de/'}
curl: (7) Failed to connect to bugzilla-next.suse.de port 443 after 14 ms: Connection refused

For me it's simply not responding again. Mentioned this on ENGINFRA-1457.

Actions #23

Updated by livdywan almost 2 years ago

  • Status changed from Workable to Blocked
Actions #24

Updated by livdywan almost 2 years ago

cdywan wrote:

For me it's simply not responding again. Mentioned this on ENGINFRA-1457.

Still waiting on this.

Actions #25

Updated by osukup almost 2 years ago

  • Status changed from Blocked to In Progress

I'm working on change of openQA_review to be able use both apis ( if is api_key in config present use rest else json-rpc )

Actions #26

Updated by livdywan almost 2 years ago

osukup wrote:

I'm working on change of openQA_review to be able use both apis ( if is api_key in config present use rest else json-rpc )

How was this unblocked now? And why does it sound like you're pursuing a very different plan now than we discussed within the team? Some reasoning would be appreciated.

Actions #27

Updated by openqa_review almost 2 years ago

  • Due date set to 2023-01-25

Setting due date based on mean cycle time of SUSE QE Tools

Actions #28

Updated by osukup almost 2 years ago

  • Status changed from In Progress to Blocked

https://suse.slack.com/archives/C02AJ1E568M/p1673437437406349?thread_ts=1673437367.328629&cid=C02AJ1E568M -> @okurz rejecting move to REST api ... so blocked until JSON-RPC is working on staging instance with token

Actions #29

Updated by livdywan almost 2 years ago

  • Due date deleted (2023-01-25)

This is blocked and we don't have any current timeline so the due date is not helpful

Actions #30

Updated by livdywan almost 2 years ago

  • Due date set to 2023-02-03
  • Status changed from Blocked to Feedback

This isn't blocking on a ticket, but we need to confirm that we have an instance to test against.

Actions #31

Updated by livdywan almost 2 years ago

  • Due date changed from 2023-02-03 to 2023-02-10

Bumping due date due to hackweek.

Actions #32

Updated by okurz almost 2 years ago

  • Due date changed from 2023-02-10 to 2023-02-24

Discussed within the weekly unblock. https://bugzilla-next.suse.de/ is currently not available, showing Error 500.

Actions #33

Updated by okurz almost 2 years ago

@cdywan @osukup https://bugzilla-next.suse.de/ is reachable right now. I recommend you crosscheck our tooling against that instance with as little effort as possible.

Actions #34

Updated by okurz almost 2 years ago

  • Status changed from Feedback to Workable
  • Priority changed from High to Urgent
Actions #35

Updated by livdywan almost 2 years ago

  • Status changed from Workable to In Progress

Ack. Checking right now

Actions #36

Updated by livdywan almost 2 years ago

For reference I'm testing with this snippet. It seems neither passing the token via headers nor params in the JSON or in the URL work... the docs say both should, and even just "token" should suffice but I'm always getting 102 for the private bug here:

for BUG in 1204372 1190670 1203578; do # non-existent public private
  echo "https://bugzilla-next.suse.de/show_bug.cgi?id=$BUG"
  curl -s -H "Bugzilla_login: $BZ_EMAIL" -H "Bugzilla_token: $BZ_API_KEY" "https://bugzilla-next.suse.de/jsonrpc.cgi?method=Bug.get&params=%5B%7B%22ids%22%3A+%5B$BUG%5D%7D%5D"
  echo .
  curl -s "https%3A%2F%2Fbugzilla-next.suse.de%2Fjsonrpc.cgi%3Fmethod%3DBug.get%26params%3D%5B%7B%22token%22%3A%22$BZ_API_KEY%22%2C%22ids%22%3A%2B%5B$BUG%5D%7D%5D"
  echo .
done

I'm also checking with Martin to confirm how the token should be passed here since it feels like the syntax may just be wrong? I expect the error won't tell me more than whether the JSON's valid.

Actions #37

Updated by livdywan almost 2 years ago

Going ahead with a draft PR anyway to ensure things are easily testable, and maybe that also helps rule out mistakes made because of manual construction of URL's: https://github.com/os-autoinst/openqa_review/pull/219

Not sure how to fix the unit tests yet.

Actions #38

Updated by livdywan almost 2 years ago

okurz wrote:

  • Check if mtui accesses bugzilla, e.g. just call git grep

With relevant code having been recently dropped I guess there's nothing to do for mtui?

Actions #39

Updated by livdywan almost 2 years ago

cdywan wrote:

Going ahead with a draft PR anyway to ensure things are easily testable, and maybe that also helps rule out mistakes made because of manual construction of URL's: https://github.com/os-autoinst/openqa_review/pull/219

Note that testing against the staging instance has one drawback. The database may not match. I solved this by locally injecting different bug numbers, i.e. something like this:

-        self.bugid = int(bugid_match.group(2)) if bugid_match else None                                                
+        self.bugid = 1203578 # 1190670

And both public and private bugs seem to work and show up in reports e.g. SKIP_SEND_MAIL=1 openqa-review -j ALP --query-issue-status (I'm using pip install -e . to run my local branch)

Actions #40

Updated by livdywan over 1 year ago

  • Assignee changed from osukup to livdywan

Maybe I should take the ticket at this point since I'm driving the changes to auto_review.

What will need to be done eventually is to add an API key to https://gitlab.suse.de/openqa/openqa-review but this isn't possible until the new version is deployed in production.

Actions #41

Updated by livdywan over 1 year ago

  • Status changed from In Progress to Feedback

https://github.com/os-autoinst/openqa_review/pull/219

Merged. Ideally nothing really changes and reports continue to be generated.

Actions #42

Updated by livdywan over 1 year ago

  • Status changed from Feedback to Blocked

This is blocking on the actual upgrade now, see Slack.

Actions #43

Updated by livdywan over 1 year ago

  • Due date deleted (2023-02-24)

This shouldn't have a due date, since it's blocked

Actions #44

Updated by okurz over 1 year ago

  • Priority changed from Urgent to Normal
Actions #45

Updated by osukup over 1 year ago

  • Related to action #127826: [tools] openqa-review CI pipelines failing added
Actions #46

Updated by osukup over 1 year ago

  • Status changed from Blocked to Resolved

new bugzilla was deployed - and Gitlab CI fixed

Actions

Also available in: Atom PDF