action #110680
coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
coordination #92854: [epic] limit overload of openQA webUI by heavy requests
Overview page shouldn't allow long-running requests without limits size:M
0%
Description
Motivation¶
To resolve #106759#note-37 the endpoint /tests/overview
needs a (reasonably high) limit on the number of jobs to prevent heartbeat and reverse proxy timeouts from being triggered.
Acceptance criteria¶
- AC1: /tests/overview requests are no longer able to block the prefork worker process for more than 400 seconds
- AC2: No limits are hard-coded (e.g. the limit can be configured in openqa.ini)
Suggestions¶
- Add a hard limit on the number of database rows
- Render an error message e.g. "Only n results included, please narrow down your search parameters"
- Use the
/tests/overview?version=Tumbleweed&groupid=24
dataset for testing
Related issues
History
#1
Updated by cdywan about 2 months ago
- Copied from action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M added
#2
Updated by okurz about 2 months ago
- Project changed from openQA Infrastructure to openQA Project
- Due date deleted (
2022-05-13) - Category set to Feature requests
#3
Updated by okurz about 2 months ago
- Parent task set to #92854
#4
Updated by kraih about 2 months ago
- Assignee set to kraih
Been some time since i worked on the frontend.
#5
Updated by cdywan about 2 months ago
- Copied from deleted (action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M)
#6
Updated by cdywan about 2 months ago
- Blocked by action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M added
#7
Updated by cdywan about 2 months ago
- Blocked by deleted (action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M)
#8
Updated by cdywan about 2 months ago
- Blocks action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M added
#9
Updated by okurz about 2 months ago
- Related to action #97190: Limit size of initial requests everywhere, e.g. /, /tests, etc. added
#10
Updated by kraih about 1 month ago
- Status changed from Workable to In Progress
#11
Updated by kraih about 1 month ago
I have replicated the issue locally with a copy of O3.
#12
Updated by openqa_review about 1 month ago
- Due date set to 2022-06-07
Setting due date based on mean cycle time of SUSE QE Tools
#13
Updated by kraih about 1 month ago
#14
Updated by kraih about 1 month ago
Side note: the actual cause for prefork worker processes blocking everything and getting killed was the query that retrieves the comments for the jobs on the /tests/overview
page.
#15
Updated by kraih about 1 month ago
- Status changed from In Progress to Feedback
PR has been merged.
#17
Updated by okurz 27 days ago
- Status changed from Resolved to Feedback
As discussed in https://suse.slack.com/archives/C02CANHLANP/p1653980932015039 https://openqa.opensuse.org/tests/overview?result=incomplete&groupid=1&groupid=3 shows "Only 500 results included, please narrow down your search parameters." even when there are zero jobs displayed
#18
Updated by kraih 27 days ago
The new problem has its own ticket now: https://progress.opensuse.org/issues/111833 #111833