action #35389
closedImprove performance of /admin/assets
Description
With ~12000 assets on openqa.suse.de the table rendering kills mortal browsers.
The HTML is 11MB and requires 50 seconds to load - which is not great, but acceptable if not the browser
would be completely unresponsive during that time.
Possibly loading the data from another route or full server side rendering - I don't know what's best,
but this deserves improvements.
Updated by mitiao over 5 years ago
- Assignee set to mitiao
this page is getting bigger and browser always lose response during loading.
investigate a better solution is needed.
Updated by szarate over 5 years ago
- Assignee deleted (
mitiao) - Priority changed from Normal to High
Updated by mkittler over 5 years ago
- Assignee set to mkittler
- Priority changed from High to Normal
I don't know what's best
I would not make it a server-side data table because those are also not so nice to use (loading time after every action like selecting a page, filtering, ...). Better just load the whole data as JSON via a separate route. The initial loading time would be longer but the table more usable. (Compared to what we have now the initial loading time would still decrease and the browser would stay responsible.)
Updated by mkittler over 5 years ago
According to my tests with a database dump from OSD using an extra JSON route would only improve the responsiveness of the page but the initial loading would still take some time.
The bottleneck is - according to nytprof - the database query and also the sorting/prioritizing of the results which is currently done in Perl code. Without sorting it takes only half of the time.
I can try to optimize that. Especially the sorting should be done by the database. However, this will be difficult because the criteria for the order is spread over multiple tables. But I'll give it a try.
Note that fast sorting would also be a prerequisite for making the table fully server-side because in this case the entire sorting must be done by the server for each user interaction with the table.
Updated by mkittler over 5 years ago
- Status changed from New to In Progress
PR: https://github.com/os-autoinst/openQA/pull/1750
The computation of the data is now faster and there are more tests. However, the table is still rendered as HTML elements and there's no caching. So the next step would be serving the data as JSON and implement caching. The cache would be refreshed when limit_assets
runs or when the user passes some parameter to force the refresh.
Serving JSON via an additional route will be a bit tricky, though. This is because the 'Assets by group' list relies on the same data but can not easily be made up from some JSON data served by another route like it is possible with the DataTable.
Updated by coolo over 5 years ago
- Target version changed from Ready to Current Sprint
- Difficulty changed from medium to hard
Updated by mkittler over 5 years ago
PR: https://github.com/os-autoinst/openQA/pull/1821
The static/cache approach works basically. Locally it takes now 10 seconds to load (including client side rendering). The code and performance could still be improved, but at least the table is usable again.
A button to re-trigger the minion job is still missing. Right now it is only possible to use the query parameter force_refresh=1
but I suppose running the asset cleanup/status algorithm only via minion would be the better option.
It is still possible to delete assets within the table. However, the change is not immediately visible anymore due to the caching. Not sure how 'bad' this is (the cleanup is automated anyways).
Updated by mkittler over 5 years ago
- Status changed from In Progress to Feedback
Let's see how fast is loads in production. If it is at least usable again I would close the issue. Making it server-side would be a further enhancement.
Updated by mkittler over 5 years ago
- Status changed from Feedback to Resolved
It takes now < 25 seconds to load in production. Since the page isn't accessed that often I suppose that's ok (and it is already a huge improvement to how slow it was before).
So I'm closing the ticket. When further improvements are required (see comments on PR), just reopen or create a new ticket.
Updated by coolo over 5 years ago
- Target version changed from Current Sprint to Done