Project

General

Profile

Actions
Copy link

action #97190

open

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

coordination #92854: [epic] limit overload of openQA webUI by heavy requests

Limit size of initial requests everywhere, e.g. /, /tests, etc., over webUI and API

Added by okurz over 3 years ago. Updated over 1 year ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
QA (public) - future
Start date:
2022-05-30
Due date:
% Done:

76%

Estimated time:
(Total: 0.00 h)

Description

Motivation

We have more and more tests, e.g. on https://openqa.suse.de/, either completed tests or sometimes even just scheduled tests, see #97043 . This showed that some routes or pages can become quite slow. IMHO a good design approach is to always limit size of initial requests, i.e. always foresee that there could be much more data in a certain set that initially imagined

Ideas

  • Wherever we process openQA jobs initially limit. We already do it for "finished" jobs on /tests but not for "running" or "scheduled", both in webUI as well as API
  • Add configurable limit for "finished", then extend to "scheduled" and "running"
  • Run (at least a single-time manual test) of 100k or more "scheduled" or "running" jobs :)
  • the index page of https://openqa.suse.de/ at time of writing takes like 10s to show results. Likely the index page should also only process less data by default if it's "too much" and load more only on request

Subtasks 15 (6 open — 9 closed)

action #111770: Limit finished tests on /tests, but query configurable and show complete number of jobs size:SResolvedmkittler2022-05-30

Actions
action #113189: Research where we need limits size:SResolvedmkittler2022-07-04

Actions
action #114421: Add a limit where it makes sense after we have it for /tests, query configurable size:MResolvedrobert.richardson

Actions
action #119428: Ensure users can get all the data for limited queries, e.g. with paginationNew2022-11-22

Actions
action #120841: Add pagination for GET /api/v1/assets size:MResolvedkraih2022-11-22

Actions
action #121048: Add pagination for GET /api/v1/bugsResolvedkraih2022-11-22

Actions
action #121099: Add pagination for GET /api/v1/jobs/<job_id:num>/commentsNew

Actions
action #121102: Add pagination for GET /api/v1/jobs size:MResolvedkraih

Actions
action #121105: Add pagination for GET /api/v1/test_suites, GET /api/v1/test_suites/:id, GET /api/v1/machines, GET api/v1/machines/:id, GET /api/v1/products, GET /api/v1/products:id size:MResolvedkraih

Actions
action #121108: Add pagination for GET /api/v1/workersResolvedkraih

Actions
action #121111: Add pagination for GET /api/v1/job_templatesNew

Actions
action #121114: Add pagination for GET /api/v1/job_settings/jobsNew

Actions
action #121117: Add pagination for GET /experimental/searchNew

Actions
action #119431: Inform users e.g. in the webUI if not all results are returned size:MWorkable2022-10-26

Actions
action #124230: "Off-by-one" error when using the limit=1 openQA API parameterResolvedmkittler2023-02-09

Actions

Related issues 3 (0 open — 3 closed)

Related to openQA Project (public) - action #41054: /tests is superslow if there are a lot of scheduled jobsResolvedmkittler2018-09-14

Actions
Related to openQA Project (public) - action #110680: Overview page shouldn't allow long-running requests without limits size:MResolvedkraih2022-02-03

Actions
Related to openQA Project (public) - action #110677: Investigation page shouldn't involve blocking long-running API routes size:MResolvedtinita2022-02-03

Actions
Actions
Copy link
 #1

Updated by okurz over 3 years ago

  • Related to action #41054: /tests is superslow if there are a lot of scheduled jobs added
Actions
Copy link
 #2

Updated by mkittler over 3 years ago

It would still be nice to see the total number of scheduled, blocked and running jobs on "All tests". So when limiting the number of rows that should be taken into account.

Note that I improved the performance of loading the table of running jobs yesterday: https://github.com/os-autoinst/openQA/pull/4133

Actions
Copy link
 #3

Updated by okurz over 3 years ago

  • Parent task set to #92854
Actions
Copy link
 #4

Updated by okurz over 3 years ago

Actions
Copy link
 #5

Updated by okurz about 3 years ago

  • Target version changed from future to Ready
Actions
Copy link
 #6

Updated by okurz about 3 years ago

  • Related to action #110680: Overview page shouldn't allow long-running requests without limits size:M added
Actions
Copy link
 #7

Updated by okurz about 3 years ago

  • Related to action #110677: Investigation page shouldn't involve blocking long-running API routes size:M added
Actions
Copy link
 #8

Updated by okurz about 3 years ago

  • Status changed from New to Blocked
  • Assignee set to okurz

blocked by #110677 and #110680

Actions
Copy link
 #9

Updated by okurz almost 3 years ago

  • Status changed from Blocked to New
  • Assignee deleted (okurz)

Both #110677 and #110680 resolved, now we can rediscuss what to do for other tables, especially /tests

Actions
Copy link
 #10

Updated by okurz almost 3 years ago

  • Subject changed from Limit size of initial requests everywhere, e.g. /, /tests, etc. to Limit size of initial requests everywhere, e.g. /, /tests, etc., over webUI and API
  • Description updated (diff)
Actions
Copy link
 #11

Updated by okurz almost 3 years ago

  • Description updated (diff)
Actions
Copy link
 #12

Updated by okurz almost 3 years ago

  • Status changed from New to Blocked
  • Assignee set to okurz
Actions
Copy link
 #13

Updated by okurz over 1 year ago

  • Status changed from Blocked to New
  • Assignee deleted (okurz)
  • Target version changed from Ready to future

All subtasks currently not in backlog, removing

Actions
Copy link

Also available in: Atom PDF