Project

General

Profile

action #90818

[openqa][tool] Not able to get group_overview json output.

Added by xlai 4 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Support
Target version:
Start date:
2021-04-08
Due date:
2021-04-22
% Done:

0%

Estimated time:
Difficulty:

Description

I meet a new issue when querying osd group_overview json format output after osd deployed the latest update this Wednesday(April 7, 2021). It had been working for long time before it. Would you please help to have a look? It blocks our openqa job retrigger tool. Thanks a lot!

Details:

  • Before update, works well, output is similar with below(did not paste original because output is very long):
    2021-04-05 01:00:01, general_utils.py, DEBUG:Command_args for subprocess to run is: ['/usr/share/openqa/script/client', '--json-output', '--host', 'http://openqa.q
    a2.suse.asia', '--apikey', 'keyxxxx', '--apisecret', 'secretxxx', '--apibase', '/', 'group_overview/20', 'limit_builds=1']
    2021-04-05 01:00:02, general_utils.py, DEBUG:Command output is: b'{\n "comments" : [],\n "build_results" : [\n {\n "escaped_build" : "162_7",\n

    "failed" : 3,\n "build" : "162.7",\n "softfailed" : 0,\n "escaped_id" : "15_SP3-162_7",\n "passed" : 116,\n "version" : "15-SP
    3",\n "escaped_version" : "15_SP3",\n "oldest" : "2021-03-15T10:51:17",\n "unfinished" : 0,\n "distris" : {\n "sle" : 1\n

    },\n "key" : "15-SP3-162.7",\n "labeled" : 0,\n "skipped" : 1,\n "total" : 120,\n "reviewed" : "",\n "all_passed" : ""\n

    }\n ],\n "max_jobs" : 120,\n "pinned_comments" : [],\n "group" : {\n "id" : 20,\n "is_parent" : null,\n "name" : "SLE-15-SP3-Performance",\n

    "rendered_description" : null\n },\n "description" : null\n}\n'
    2021-04-05 01:00:02, general_utils.py, DEBUG:After json load, data is {'comments': [], 'build_results': [{'escaped_build': '162_7', 'failed': 3, 'build': '162.7',
    'softfailed': 0, 'escaped_id': '15_SP3-162_7', 'passed': 116, 'version': '15-SP3', 'escaped_version': '15_SP3', 'oldest': '2021-03-15T10:51:17', 'unfinished': 0, 'distris':
    {'sle': 1}, 'key': '15-SP3-162.7', 'labeled': 0, 'skipped': 1, 'total': 120, 'reviewed': '', 'all_passed': ''}], 'max_jobs': 120, 'pinned_comments': [], 'group': {'id': 20
    , 'is_parent': None, 'name': 'SLE-15-SP3-Performance', 'rendered_description': None}, 'description': None}
    2021-04-05 01:00:02, openqa_job_retrigger.py, INFO:Group 20's latest build is 162.7, will handle this build.

  • After update, return reports error:
    qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.suse.de --apikey --apisecret --apibase / group_overview/263 limit_builds=1

    hash- or arrayref expected (not a simple scalar, use allow_nonref to allow this) at /usr/share/openqa/script/client line 174.
    qa2-dhcp-53:~ #


Related issues

Related to openQA Project - action #91004: Treat '--apibase /' argument for openqa-client as errorWorkable2021-04-12

History

#1 Updated by Xiaojing_liu 4 months ago

I also did some tests in local, have different result:

./client --json-output --host https://openqa.suse.de --apibase / group_overview/263 limit_builds=1
null

I used /openqa-cli api --osd --apibase / group_overview/263 limit_builds=1 --json, it could get the results, but the format is html. not a json data.

I guess the related pr is: https://github.com/os-autoinst/openQA/pull/3820

#2 Updated by okurz 4 months ago

  • Category set to Support
  • Assignee set to okurz
  • Target version set to Ready

#4 Updated by mkittler 4 months ago

However, since 2 PRs to handle it have been merged, it shouldn't be a problem anymore. Accessing e.g. https://openqa.suse.de/tests/overview.json?distri=sle&version=15-SP3&build=172.3&groupid=110 within the browser also returns JSON. The following works as well:

openqa-cli api --osd --apibase / group_overview/263.json limit_builds=1

So just be sure to use the .json extension.

(Note that --json is for setting the content type if you want to upload a JSON document and shouldn't influence the response.)

#5 Updated by kraih 4 months ago

mkittler wrote:

openqa-cli api --osd --apibase / group_overview/263.json limit_builds=1

So just be sure to use the .json extension.

Actually, the .json extension should not even be necessary with openqa-cli api. It always sets the Accept: application/json header (as long as you don't override it with your own Accept header).

#6 Updated by kraih 4 months ago

kraih wrote:

Actually, the .json extension should not even be necessary with openqa-cli api. It always sets the Accept: application/json header (as long as you don't override it with your own Accept header).

I was wrong with that unfortunately. The workaround we use in the routes always sets the default extension to .html, which make the Accept header do nothing. So content negotiation is kinda broken at the moment.

#7 Updated by mkittler 4 months ago

Is this something that could be addressed on Mojolicious-level (assuming openQA's code is supposed to work)?

#8 Updated by kraih 4 months ago

Only solution i can think of would require us to depend on Mojolicious 9.11+. I've already prepared a PR. https://github.com/os-autoinst/openQA/pull/3830

#9 Updated by okurz 4 months ago

  • Due date set to 2021-04-22
  • Status changed from New to Feedback

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all but instead retrieve the web pages that are meant for consumption by browsers and human readers. If you want to have a machine readable representation of the "group overview" page in JSON format you can use for example

curl -s https://openqa.opensuse.org/group_overview/1.json

Both openqa-client and openqa-cli are not meant and not necessary to access these pages. Good enough?

#10 Updated by cdywan 4 months ago

Shouldn't --apibase / actually cause an error? It makes no sense to point it at anything that's not openQA-compatible RESTful API. At least I don't see what the purpose of that would be

#11 Updated by xlai 4 months ago

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

but instead retrieve the web pages that are meant for consumption by browsers and human readers. If you want to have a machine readable representation of the "group overview" page in JSON format you can use for example

curl -s https://openqa.opensuse.org/group_overview/1.json

Both openqa-client and openqa-cli are not meant and not necessary to access these pages. Good enough?

okurz Thanks for the comment.

I can switch to use /usr/share/openqa/script/client ... group_overview/1.json, and it works in my environment.

#12 Updated by cdywan 4 months ago

xlai wrote:

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

The default is /api/v1 which is the RESTful openQA API. This is why I'm surprised it doesn't cause an error - this is not just a path

#13 Updated by xlai 4 months ago

cdywan wrote:

xlai wrote:

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

The default is /api/v1 which is the RESTful openQA API. This is why I'm surprised it doesn't cause an error - this is not just a path

cdywan But why from my trying, it does not work to use apibase /api/v1(I hide my api key/secret when paste)?

qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.nue.suse.com --apikey --apisecret --apibase /api/v1 group_overview/263 limit_builds=1
ERROR: 404 - Not Found

{ error_status => 404 }

qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.nue.suse.com --apikey --apisecret --apibase /api/v1 group_overview/263

ERROR: 404 - Not Found

{ error_status => 404 }

qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com ---apikey --apisecret --apibase /api/v1 group_overview/263
ERROR: 404 - Not Found

{ error_status => 404 }
qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com --apikey --apisecret --apibase api/v1 group_overview/263
ERROR: 404 - Not Found
{ error_status => 404 }
qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com --apikey --apisecret --apibase api/v1 group_overview/263.json
ERROR: 404 - Not Found
{ error_status => 404 }
qa2-dhcp-53:~ #

#14 Updated by okurz 4 months ago

xlai wrote:

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

There is none. Because the group overview in the same way you want to get it is not an "API call" per se but a JSON representation of the html page which you can get with curl in the way I described. No need to use the openqa-client.

xlai wrote:

cdywan wrote:

xlai wrote:

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

The default is /api/v1 which is the RESTful openQA API. This is why I'm surprised it doesn't cause an error - this is not just a path

cdywan But why from my trying, it does not work to use apibase /api/v1(I hide my api key/secret when paste)?

qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.nue.suse.com --apikey --apisecret --apibase /api/v1 group_overview/263 limit_builds=1
ERROR: 404 - Not Found

{ error_status => 404 }

qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.nue.suse.com --apikey --apisecret --apibase /api/v1 group_overview/263

ERROR: 404 - Not Found

{ error_status => 404 }

qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com ---apikey --apisecret --apibase /api/v1 group_overview/263
ERROR: 404 - Not Found

{ error_status => 404 }
qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com --apikey --apisecret --apibase api/v1 group_overview/263
ERROR: 404 - Not Found
{ error_status => 404 }
qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com --apikey --apisecret --apibase api/v1 group_overview/263.json
ERROR: 404 - Not Found
{ error_status => 404 }
qa2-dhcp-53:~ #

Because these API routes do not exist. https://openqa.nue.suse.com/what_are_supported_api_routes lists all existing paths for both the real API routes (below /api/v1) as well as the ones that are used internally as well as those pages that can be accessed directly (e.g. /tests).

Further hints:

  • Put your API keys and secrets in ~/.config/openqa/client.conf, do not append apikey and apisecret parameter. Then you are also not running into risk into copy-pasting them into a ticket comment :)
  • apibase "/api/v1" is the implicit default. Better don't use the apibase parameter, there is currently no good use case for that. Likely this parameter was introduced to allow switching to a different API version but so far we have only a single one
  • Use openqa-client from PATH instead of a full path /usr/share/openqa/script/client
  • Consider using openqa-cli api --osd instead of openqa-client --host http://openqa.nue.suse.com

#15 Updated by xlai 4 months ago

okurz, thanks for the detailed reply.

#16 Updated by xlai 4 months ago

  • Status changed from Feedback to Resolved

#17 Updated by okurz 4 months ago

  • Related to action #91004: Treat '--apibase /' argument for openqa-client as error added

Also available in: Atom PDF