Project

General

Profile

action #120276

coordination #120522: [epic] Support upgrade SUSE bugzilla instance

Prepare openqa_bugfetcher for upcoming bugzilla update

Added by dheidler 3 months ago. Updated 13 days ago.

Status:
Blocked
Priority:
Normal
Assignee:
Target version:
Start date:
2022-11-10
Due date:
% Done:

0%

Estimated time:

Description

Observation

Hello QE,

first I want to express my gratitude to Chenzi Cao for huge contribution to Bugzilla UAT testing. All reported bugs were fixed, however there are still four issues to be resolved which block deployment to production. Currently the go live is planned for 6.12.-8.12.2022, but there might be further delay.

One of the big changes is, that XML-RPC support will be dropped and replaced by REST api described here https://bugzilla.readthedocs.io/en/5.0.4/api/ . This enables implementation of several QE metrics identified in the past here: https://confluence.suse.com/display/qasle/QA+Metrics

If there is someone, who would be interested in working on ingesting the data form Bugzilla, processing it according to identified metrics and storing in InfludDB in order to display it in Grafana, please let me know. It could be an interesting and fun project which would make lives of many people easier.

Have a nice day

Jan

openqa_bugfetcher is currently using XML-RPC (actually JSON-RPC) to fetch bug information.
There is a fallback to fetch at least basic information via HTML scraping but that only works for public bugs and might break as well for when there is a major update applied.

https://github.com/os-autoinst/openqa_bugfetcher
https://github.com/os-autoinst/openqa_bugfetcher/blob/master/openqa_bugfetcher/issues/bugzilla_issue.py

After updating the sources, make sure to update the RPM at https://build.opensuse.org/package/show/devel:openQA/openqa_bugfetcher
and install the update on the openqa-service VM.

Suggestions


Related issues

Related to QA - action #120525: Ensure our usual bugzilla integrated tooling works with the upgraded test instance size:MFeedback2022-11-152023-02-10

Copied to openQA Infrastructure - action #120279: Proper maintainership for openqa_service VM size:MWorkable2022-11-10

History

#1 Updated by okurz 3 months ago

  • Priority changed from Normal to High
  • Target version set to Ready

For the user acceptance test we stated that we need jsonrpc. We should clarify if that still works, if it can be enabled or if we need to change to REST.

And I think we need to talk about maintainership of the openqa_service VM, e.g. updates, upgrades, backup, config management, monitoring, etc.

#2 Updated by okurz 3 months ago

  • Copied to action #120279: Proper maintainership for openqa_service VM size:M added

#3 Updated by cdywan 3 months ago

  • Subject changed from Prepare openqa_bugfetcher for upcoming bugzilla update to Prepare openqa_bugfetcher for upcoming bugzilla update size:M
  • Description updated (diff)
  • Status changed from New to Feedback
  • Assignee set to dheidler

Setting to Feedback with dheidler as the assignee since a question was already posted on the email from Jan

#4 Updated by jstehlik 3 months ago

Thanks for creating this ticket and making sure that openqa_bugfetcher still works.
I asked Johannes van Wijk in #proj-it-bugzilla-improvement if JSON-RPC will work.

My guess is that not, because the reason for deprecation is authentication. "In the staging version there was still the option to use the XML-RPC 'api', because the IDP was still not implemented, but with the implementation of the IDP solution, we came unfortunately to the conclusion that we will the option to use the XML-RPC. , because the authentication method is no longer supported / depreciated as of version 5.0.4"

Will confirm as soon as I have the reply.

#5 Updated by jstehlik 3 months ago

The JSON-RPC API will receive no further updates and will be removed. This means migration to REST API is inevitable.

#6 Updated by okurz 3 months ago

jstehlik wrote:

The JSON-RPC API will receive no further updates and will be removed. This means migration to REST API is inevitable.

jstehlik regarding the new bugzilla version. In #113255 we clearly stated that we need the JSON-RPC interface. According to what Dominik Heidler told me know that is planned to be removed which is a major issue. Can you confirm that our bugzilla test cases have been conducted?

#8 Updated by okurz 3 months ago

  • Parent task set to #120522

#9 Updated by okurz 3 months ago

okurz wrote:

jstehlik regarding the new bugzilla version. In #113255 we clearly stated that we need the JSON-RPC interface. According to what Dominik Heidler told me know that is planned to be removed which is a major issue. Can you confirm that our bugzilla test cases have been conducted?

Will be handled in #120522

#10 Updated by dheidler 3 months ago

REST API docs can be found here: https://bugzilla.readthedocs.io/en/5.0.4/api/

But it seems that the REST API is not yet available, so we need to wait for the update and will have a broken tool for a few days until it is modified to use the new API.

Unless there is already a new (test?) bugzilla instance that we can use to develop against.

#11 Updated by tinita 3 months ago

#12 Updated by dheidler 3 months ago

  • Status changed from Feedback to In Progress

#13 Updated by jstehlik 3 months ago

I hope we clarified it now sufficiently in #proj-it-bugzilla-improvement . Sorry if I created a unnecessary sense of urgency. The JSON RPC shall work after upgrade, but its version 1.1 will be deprecated in future. Thus JSON RPC 1.0 is recommended by Bugzilla team and at some point we will have to move to REST API. Ivan Lausuch is currently working on a python library for integration with Bugzilla and Progress. I hope it could be helpful also in the future for bugfetcher.

#14 Updated by openqa_review 3 months ago

  • Due date set to 2022-11-30

Setting due date based on mean cycle time of SUSE QE Tools

#16 Updated by okurz 3 months ago

  • Subject changed from Prepare openqa_bugfetcher for upcoming bugzilla update size:M to Prepare openqa_bugfetcher for upcoming bugzilla update

I realized that we "estimated" this but do not even have ACs

dheidler your PR in general looks ok but as we clarified we will still have JSONRPC for the foreseeable future so I suggest to only check the current implementation with changed authentication approach against the next instance, not change to REST.

#17 Updated by dheidler 3 months ago

  • Status changed from In Progress to Feedback

The PR is already done - and the rest API is easy to use.
So I see no reason to build up technical debt by sticking with the deprecated API.

Waiting for new bugzilla to deploy to merge this PR after testing.

#18 Updated by okurz 3 months ago

  • Related to action #120525: Ensure our usual bugzilla integrated tooling works with the upgraded test instance size:M added

#19 Updated by okurz 3 months ago

  • Due date deleted (2022-11-30)
  • Status changed from Feedback to Blocked

As discussed in weekly estimation we should do #120525 first

#20 Updated by okurz about 2 months ago

  • Priority changed from High to Normal

We discussed the current status in SUSE QE Tools team daily 2022-12-19. The production bugzilla instance has not been upgraded yet. We should await that and then merge https://github.com/os-autoinst/openqa_bugfetcher/pull/7 . dheidler already tested that authentication works against the test instance.
The status about the test instance can also be followed in
https://app.slack.com/client/T02863RC2AC/C02R5HRKD1C

#21 Updated by cdywan 13 days ago

Just to clarify. This is effectively blocking on #120525#note-28 because the staging instance still isn't reliable. We did look into the new API again but it was raising more questions. Until this is sorted out there's unfortunately not much we can do from our side.

Also available in: Atom PDF