action #110680
closedcoordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
coordination #92854: [epic] limit overload of openQA webUI by heavy requests
Overview page shouldn't allow long-running requests without limits size:M
Description
Motivation¶
To resolve #106759#note-37 the endpoint /tests/overview needs a (reasonably high) limit on the number of jobs to prevent heartbeat and reverse proxy timeouts from being triggered.
Acceptance criteria¶
- AC1: /tests/overview requests are no longer able to block the prefork worker process for more than 400 seconds
- AC2: No limits are hard-coded (e.g. the limit can be configured in openqa.ini)
Suggestions¶
- Add a hard limit on the number of database rows
- Render an error message e.g. "Only n results included, please narrow down your search parameters"
- Use the
/tests/overview?version=Tumbleweed&groupid=24dataset for testing
Updated by livdywan over 2 years ago
- Copied from action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M added
Updated by okurz over 2 years ago
- Project changed from openQA Infrastructure (public) to openQA Project (public)
- Due date deleted (
2022-05-13) - Category set to Feature requests
Updated by kraih over 2 years ago
- Assignee set to kraih
Been some time since i worked on the frontend.
Updated by livdywan over 2 years ago
- Copied from deleted (action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M)
Updated by livdywan over 2 years ago
- Blocked by action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M added
Updated by livdywan over 2 years ago
- Blocked by deleted (action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M)
Updated by livdywan over 2 years ago
- Blocks action #106759: Worker xyz has no heartbeat (400 seconds), restarting repeatedly reported on o3 size:M added
Updated by okurz over 2 years ago
- Related to action #97190: Limit size of initial requests everywhere, e.g. /, /tests, etc., over webUI and API added
Updated by kraih over 2 years ago
I have replicated the issue locally with a copy of O3.
Updated by openqa_review over 2 years ago
- Due date set to 2022-06-07
Setting due date based on mean cycle time of SUSE QE Tools
Updated by kraih over 2 years ago
Updated by kraih over 2 years ago
Side note: the actual cause for prefork worker processes blocking everything and getting killed was the query that retrieves the comments for the jobs on the /tests/overview page.
Updated by okurz over 2 years ago
- Status changed from Resolved to Feedback
As discussed in https://suse.slack.com/archives/C02CANHLANP/p1653980932015039 https://openqa.opensuse.org/tests/overview?result=incomplete&groupid=1&groupid=3 shows "Only 500 results included, please narrow down your search parameters." even when there are zero jobs displayed
Updated by kraih over 2 years ago
The new problem has its own ticket now: https://progress.opensuse.org/issues/111833 #111833