Added by coolo about 6 years ago. Updated over 5 years ago.
Description
With ~12000 assets on openqa.suse.de the table rendering kills mortal browsers.
The HTML is 11MB and requires 50 seconds to load - which is not great, but acceptable if not the browser
would be completely unresponsive during that time.
Possibly loading the data from another route or full server side rendering - I don't know what's best,
but this deserves improvements.
this page is getting bigger and browser always lose response during loading.
investigate a better solution is needed.
I don't know what's best
I would not make it a server-side data table because those are also not so nice to use (loading time after every action like selecting a page, filtering, ...). Better just load the whole data as JSON via a separate route. The initial loading time would be longer but the table more usable. (Compared to what we have now the initial loading time would still decrease and the browser would stay responsible.)
According to my tests with a database dump from OSD using an extra JSON route would only improve the responsiveness of the page but the initial loading would still take some time.
The bottleneck is - according to nytprof - the database query and also the sorting/prioritizing of the results which is currently done in Perl code. Without sorting it takes only half of the time.
I can try to optimize that. Especially the sorting should be done by the database. However, this will be difficult because the criteria for the order is spread over multiple tables. But I'll give it a try.
Note that fast sorting would also be a prerequisite for making the table fully server-side because in this case the entire sorting must be done by the server for each user interaction with the table.
PR: https://github.com/os-autoinst/openQA/pull/1750
The computation of the data is now faster and there are more tests. However, the table is still rendered as HTML elements and there's no caching. So the next step would be serving the data as JSON and implement caching. The cache would be refreshed when limit_assets runs or when the user passes some parameter to force the refresh.
Serving JSON via an additional route will be a bit tricky, though. This is because the 'Assets by group' list relies on the same data but can not easily be made up from some JSON data served by another route like it is possible with the DataTable.
PR: https://github.com/os-autoinst/openQA/pull/1821
The static/cache approach works basically. Locally it takes now 10 seconds to load (including client side rendering). The code and performance could still be improved, but at least the table is usable again.
A button to re-trigger the minion job is still missing. Right now it is only possible to use the query parameter force_refresh=1 but I suppose running the asset cleanup/status algorithm only via minion would be the better option.
It is still possible to delete assets within the table. However, the change is not immediately visible anymore due to the caching. Not sure how 'bad' this is (the cleanup is automated anyways).
Let's see how fast is loads in production. If it is at least usable again I would close the issue. Making it server-side would be a further enhancement.
It takes now < 25 seconds to load in production. Since the page isn't accessed that often I suppose that's ok (and it is already a huge improvement to how slow it was before).
So I'm closing the ticket. When further improvements are required (see comments on PR), just reopen or create a new ticket.