Project

General

Profile

Actions
Copy link

action #90818

closed

[openqa][tool] Not able to get group_overview json output.

Added by xlai over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Category:
Support
Target version:
Ready
Start date:
2021-04-08
Due date:
2021-04-22
% Done:

0%

Estimated time:

Description

I meet a new issue when querying osd group_overview json format output after osd deployed the latest update this Wednesday(April 7, 2021). It had been working for long time before it. Would you please help to have a look? It blocks our openqa job retrigger tool. Thanks a lot!

Details:

  • Before update, works well, output is similar with below(did not paste original because output is very long):
    2021-04-05 01:00:01, general_utils.py, DEBUG:Command_args for subprocess to run is: ['/usr/share/openqa/script/client', '--json-output', '--host', 'http://openqa.q
    a2.suse.asia', '--apikey', 'keyxxxx', '--apisecret', 'secretxxx', '--apibase', '/', 'group_overview/20', 'limit_builds=1']
    2021-04-05 01:00:02, general_utils.py, DEBUG:Command output is: b'{\n "comments" : [],\n "build_results" : [\n {\n "escaped_build" : "162_7",\n

    "failed" : 3,\n "build" : "162.7",\n "softfailed" : 0,\n "escaped_id" : "15_SP3-162_7",\n "passed" : 116,\n "version" : "15-SP
    3",\n "escaped_version" : "15_SP3",\n "oldest" : "2021-03-15T10:51:17",\n "unfinished" : 0,\n "distris" : {\n "sle" : 1\n

    },\n "key" : "15-SP3-162.7",\n "labeled" : 0,\n "skipped" : 1,\n "total" : 120,\n "reviewed" : "",\n "all_passed" : ""\n

    }\n ],\n "max_jobs" : 120,\n "pinned_comments" : [],\n "group" : {\n "id" : 20,\n "is_parent" : null,\n "name" : "SLE-15-SP3-Performance",\n

    "rendered_description" : null\n },\n "description" : null\n}\n'
    2021-04-05 01:00:02, general_utils.py, DEBUG:After json load, data is {'comments': [], 'build_results': [{'escaped_build': '162_7', 'failed': 3, 'build': '162.7',
    'softfailed': 0, 'escaped_id': '15_SP3-162_7', 'passed': 116, 'version': '15-SP3', 'escaped_version': '15_SP3', 'oldest': '2021-03-15T10:51:17', 'unfinished': 0, 'distris':
    {'sle': 1}, 'key': '15-SP3-162.7', 'labeled': 0, 'skipped': 1, 'total': 120, 'reviewed': '', 'all_passed': ''}], 'max_jobs': 120, 'pinned_comments': [], 'group': {'id': 20
    , 'is_parent': None, 'name': 'SLE-15-SP3-Performance', 'rendered_description': None}, 'description': None}
    2021-04-05 01:00:02, openqa_job_retrigger.py, INFO:Group 20's latest build is 162.7, will handle this build.

  • After update, return reports error:
    qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.suse.de --apikey --apisecret --apibase / group_overview/263 limit_builds=1

    hash- or arrayref expected (not a simple scalar, use allow_nonref to allow this) at /usr/share/openqa/script/client line 174.
    qa2-dhcp-53:~ #

Related issues 1 (1 open — 0 closed)

Related to openQA Project (public) - action #91004: Treat '--apibase /' argument for openqa-client as errorWorkable2021-04-12

Actions
Actions
Copy link
 #1

Updated by Xiaojing_liu over 3 years ago

I also did some tests in local, have different result:

./client --json-output --host https://openqa.suse.de --apibase / group_overview/263 limit_builds=1
null

I used /openqa-cli api --osd --apibase / group_overview/263 limit_builds=1 --json, it could get the results, but the format is html. not a json data.

I guess the related pr is: https://github.com/os-autoinst/openQA/pull/3820

Actions
Copy link
 #2

Updated by okurz over 3 years ago

  • Category set to Support
  • Assignee set to okurz
  • Target version set to Ready
Actions
Copy link
 #4

Updated by mkittler over 3 years ago

However, since 2 PRs to handle it have been merged, it shouldn't be a problem anymore. Accessing e.g. https://openqa.suse.de/tests/overview.json?distri=sle&version=15-SP3&build=172.3&groupid=110 within the browser also returns JSON. The following works as well:

openqa-cli api --osd --apibase / group_overview/263.json limit_builds=1

So just be sure to use the .json extension.

(Note that --json is for setting the content type if you want to upload a JSON document and shouldn't influence the response.)

Actions
Copy link
 #5

Updated by kraih over 3 years ago

mkittler wrote:

openqa-cli api --osd --apibase / group_overview/263.json limit_builds=1

So just be sure to use the .json extension.

Actually, the .json extension should not even be necessary with openqa-cli api. It always sets the Accept: application/json header (as long as you don't override it with your own Accept header).

Actions
Copy link
 #6

Updated by kraih over 3 years ago

kraih wrote:

Actually, the .json extension should not even be necessary with openqa-cli api. It always sets the Accept: application/json header (as long as you don't override it with your own Accept header).

I was wrong with that unfortunately. The workaround we use in the routes always sets the default extension to .html, which make the Accept header do nothing. So content negotiation is kinda broken at the moment.

Actions
Copy link
 #7

Updated by mkittler over 3 years ago

Is this something that could be addressed on Mojolicious-level (assuming openQA's code is supposed to work)?

Actions
Copy link
 #8

Updated by kraih over 3 years ago

Only solution i can think of would require us to depend on Mojolicious 9.11+. I've already prepared a PR. https://github.com/os-autoinst/openQA/pull/3830

Actions
Copy link
 #9

Updated by okurz over 3 years ago

  • Due date set to 2021-04-22
  • Status changed from New to Feedback

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all but instead retrieve the web pages that are meant for consumption by browsers and human readers. If you want to have a machine readable representation of the "group overview" page in JSON format you can use for example

curl -s https://openqa.opensuse.org/group_overview/1.json

Both openqa-client and openqa-cli are not meant and not necessary to access these pages. Good enough?

Actions
Copy link
 #10

Updated by livdywan over 3 years ago

Shouldn't --apibase / actually cause an error? It makes no sense to point it at anything that's not openQA-compatible RESTful API. At least I don't see what the purpose of that would be

Actions
Copy link
 #11

Updated by xlai over 3 years ago

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

but instead retrieve the web pages that are meant for consumption by browsers and human readers. If you want to have a machine readable representation of the "group overview" page in JSON format you can use for example

curl -s https://openqa.opensuse.org/group_overview/1.json

Both openqa-client and openqa-cli are not meant and not necessary to access these pages. Good enough?

@okurz Thanks for the comment.

I can switch to use /usr/share/openqa/script/client ... group_overview/1.json, and it works in my environment.

Actions
Copy link
 #12

Updated by livdywan over 3 years ago

xlai wrote:

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

The default is /api/v1 which is the RESTful openQA API. This is why I'm surprised it doesn't cause an error - this is not just a path

Actions
Copy link
 #13

Updated by xlai over 3 years ago

cdywan wrote:

xlai wrote:

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

The default is /api/v1 which is the RESTful openQA API. This is why I'm surprised it doesn't cause an error - this is not just a path

@cdywan But why from my trying, it does not work to use apibase /api/v1(I hide my api key/secret when paste)?

qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.nue.suse.com --apikey --apisecret --apibase /api/v1 group_overview/263 limit_builds=1
ERROR: 404 - Not Found

{ error_status => 404 }

qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.nue.suse.com --apikey --apisecret --apibase /api/v1 group_overview/263

ERROR: 404 - Not Found

{ error_status => 404 }

qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com ---apikey --apisecret --apibase /api/v1 group_overview/263
ERROR: 404 - Not Found

{ error_status => 404 }
qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com --apikey --apisecret --apibase api/v1 group_overview/263
ERROR: 404 - Not Found
{ error_status => 404 }
qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com --apikey --apisecret --apibase api/v1 group_overview/263.json
ERROR: 404 - Not Found
{ error_status => 404 }
qa2-dhcp-53:~ #

Actions
Copy link
 #14

Updated by okurz over 3 years ago

xlai wrote:

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

There is none. Because the group overview in the same way you want to get it is not an "API call" per se but a JSON representation of the html page which you can get with curl in the way I described. No need to use the openqa-client.

xlai wrote:

cdywan wrote:

xlai wrote:

okurz wrote:

Basically the same as in #88363 applies:

So what you are doing with the parameter --apibase / is basically not using the intended web API at all

What is the intended web api(restapi) that client script supports to get group overview json format output?

The default is /api/v1 which is the RESTful openQA API. This is why I'm surprised it doesn't cause an error - this is not just a path

@cdywan But why from my trying, it does not work to use apibase /api/v1(I hide my api key/secret when paste)?

qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.nue.suse.com --apikey --apisecret --apibase /api/v1 group_overview/263 limit_builds=1
ERROR: 404 - Not Found

{ error_status => 404 }

qa2-dhcp-53:~ # /usr/share/openqa/script/client --json-output --host http://openqa.nue.suse.com --apikey --apisecret --apibase /api/v1 group_overview/263

ERROR: 404 - Not Found

{ error_status => 404 }

qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com ---apikey --apisecret --apibase /api/v1 group_overview/263
ERROR: 404 - Not Found

{ error_status => 404 }
qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com --apikey --apisecret --apibase api/v1 group_overview/263
ERROR: 404 - Not Found
{ error_status => 404 }
qa2-dhcp-53:~ # /usr/share/openqa/script/client --host http://openqa.nue.suse.com --apikey --apisecret --apibase api/v1 group_overview/263.json
ERROR: 404 - Not Found
{ error_status => 404 }
qa2-dhcp-53:~ #

Because these API routes do not exist. https://openqa.nue.suse.com/what_are_supported_api_routes lists all existing paths for both the real API routes (below /api/v1) as well as the ones that are used internally as well as those pages that can be accessed directly (e.g. /tests).

Further hints:

  • Put your API keys and secrets in ~/.config/openqa/client.conf, do not append apikey and apisecret parameter. Then you are also not running into risk into copy-pasting them into a ticket comment :)
  • apibase "/api/v1" is the implicit default. Better don't use the apibase parameter, there is currently no good use case for that. Likely this parameter was introduced to allow switching to a different API version but so far we have only a single one
  • Use openqa-client from PATH instead of a full path /usr/share/openqa/script/client
  • Consider using openqa-cli api --osd instead of openqa-client --host http://openqa.nue.suse.com
Actions
Copy link
 #15

Updated by xlai over 3 years ago

@okurz, thanks for the detailed reply.

Actions
Copy link
 #16

Updated by xlai over 3 years ago

  • Status changed from Feedback to Resolved
Actions
Copy link
 #17

Updated by okurz over 3 years ago

  • Related to action #91004: Treat '--apibase /' argument for openqa-client as error added
Actions
Copy link

Also available in: Atom PDF