action #120315
closedopenqa-client does not get complete asset list size:S
Description
Observation¶
The test queries openQA for the existing assets, using the command:
openqa-client --host https://openqa.opensuse.org assets get | sed -n 's/.name.*Tumbleweed-NET-x86_64-Snapshot\([0-9]\+\)-Media.\$/\1/p'
But since recently (apparently today, the test passed yesterday), ew don't get the complete list anymore:
openqa-client --host https://openqa.opensuse.org assets get | grep -c "name =" => 500
openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-openqa_bootstrap@64bit fails in
start_test
Test suite description¶
Maintainer: dheidler. Install openQA using openqa-bootstrap script.
Reproducible¶
Fails since (at least) Build 20221110
Expected result¶
Last good: 20221109 (or more recent)
Further details¶
Always latest result in this scenario: latest
Updated by okurz about 2 years ago
- Project changed from openQA Tests to openQA Project
- Category changed from Bugs in existing tests to Regressions/Crashes
- Priority changed from Normal to Urgent
- Target version set to Ready
Updated by okurz about 2 years ago
Either the test is adjusted to query more assets or we bump the default limit in openQA assuming that the list of assets isn't too verbose nor expensive
Updated by coolo about 2 years ago
It shouldn't 500, but expanding the limit sounds backwards. Supporting a filter sounds what this use case wants.
Updated by coolo about 2 years ago
Ah, I thought the HTTP status code is 500. That's an unfortunate choice of default limit :)
Updated by dimstar about 2 years ago
okurz wrote:
Either the test is adjusted to query more assets or we bump the default limit in openQA assuming that the list of assets isn't too verbose nor expensive
openqa-client --host https://openqa.opensuse.org assets get limit=100000 | grep -c "name ="
5000
Interestingly, when passing a limit to the get, the number increases but seems to hit a different limiter (the relevant assets are not in the output yet)
Updated by mkittler about 2 years ago
- Assignee set to mkittler
The 2nd limit is so no one can easily overload the server with requests that would run into a gateway timeout anyways. Of course both limits were way too low. I suppose we should have a distinct set of limits for assets that are significantly higher.
Updated by mkittler about 2 years ago
I've been changing the default limit to 10000 and you can go up to 50000 with the limit parameter. (The changes are done in /etc/openqa/openqa.ini, you can change it yourself if you need to.)
Updated by livdywan about 2 years ago
- Related to action #114421: Add a limit where it makes sense after we have it for /tests, query configurable size:M added
Updated by mkittler about 2 years ago
- Status changed from New to In Progress
PR to allow configuring this separately for assets: https://github.com/os-autoinst/openQA/pull/4896
I see you're grepping on the whole list to find an asset by its name. I suppose adding a filter/search parameter for that would be helpful. What are you searching for specifically? Is an exact match for the asset name sufficient or do you need globbing?
Updated by mkittler about 2 years ago
To answer my last question: The test does a regex match:
last_tw_build=\$(openqa-client --host $openqa_url assets get | sed -n 's/^.*name.*Tumbleweed-NET-$arch-Snapshot\\([0-9]\\+\\)-Media.*\$/\\1/p' | sort -n | tail -n 1)
echo "Last Tumbleweed build on openqa.opensuse.org: \$last_tw_build"
This could be easily implemented server-side.
Updated by livdywan about 2 years ago
- Copied to action #120336: Improve scalability of assets listing (API route) size:M added
Updated by livdywan about 2 years ago
- Subject changed from openqa-client does not get complete asset list to openqa-client does not get complete asset list size:S
Updated by mkittler about 2 years ago
- Status changed from In Progress to Resolved
@cdywan created a ticket for further improvements so I'll keep it at the mentioned PR for the sake of this ticket. It has already been merged and deployed on o3. I've just reverted my config changes from #120315#note-7 and restarted the service. Now all assets are returned again:
openqa=# select count(*) from assets;
count
-------
15645
(1 Zeile)
$ openqa-cli api --host https://openqa.opensuse.org assets | jq '.assets | length'
15645
Updated by dzedro about 2 years ago
Now I can't find tests on osd.
All from 501th will now be not shown ?
https://suse.slack.com/archives/C02CANHLANP/p1668584467225009
Updated by dzedro about 2 years ago
Another thing, how many jobs are now scheduled ?
ALso I can't see jobs from 501th on the page.
500 scheduled jobs (85 blocked by other jobs)
Updated by livdywan about 2 years ago
dzedro wrote:
Now I can't find tests on osd.
All from 501th will now be not shown ?
https://suse.slack.com/archives/C02CANHLANP/p1668584467225009
As a workaround I added [misc_limits] generic_default_limit = 5000 followed by sudo systemctl restart openqa-webui. The proper solution will be implemented as part of #19428.
Updated by okurz about 2 years ago
- Status changed from Resolved to In Progress
- Assignee changed from mkittler to okurz
Preparing more limit-bumps to handle those regressions
Updated by okurz about 2 years ago
Updated by okurz about 2 years ago
Updated by kraih almost 2 years ago
- Related to action #120841: Add pagination for GET /api/v1/assets size:M added
Updated by okurz almost 2 years ago
- Status changed from Feedback to Resolved
https://github.com/os-autoinst/openQA/pull/4912 merged and deployed to OSD means that we can remove the manual override in /etc/openqa/openqa.ini. I removed the line and restarted the webUI. I can confirm that https://openqa.suse.de/admin/test_suites still shows 2.8k entries so all good.