action #35389

Improve performance of /admin/assets

Added by coolo almost 2 years ago. Updated about 1 year ago.

Status:ResolvedStart date:24/04/2018
Priority:NormalDue date:
Assignee:mkittler% Done:

0%

Category:Feature requests
Target version:Done
Difficulty:hard
Duration:

Description

With ~12000 assets on openqa.suse.de the table rendering kills mortal browsers.

The HTML is 11MB and requires 50 seconds to load - which is not great, but acceptable if not the browser
would be completely unresponsive during that time.

Possibly loading the data from another route or full server side rendering - I don't know what's best,
but this deserves improvements.

History

#1 Updated by mitiao over 1 year ago

  • Assignee set to mitiao

this page is getting bigger and browser always lose response during loading.
investigate a better solution is needed.

#2 Updated by szarate over 1 year ago

  • Assignee deleted (mitiao)
  • Priority changed from Normal to High

#3 Updated by mkittler over 1 year ago

  • Assignee set to mkittler
  • Priority changed from High to Normal

I don't know what's best

I would not make it a server-side data table because those are also not so nice to use (loading time after every action like selecting a page, filtering, ...). Better just load the whole data as JSON via a separate route. The initial loading time would be longer but the table more usable. (Compared to what we have now the initial loading time would still decrease and the browser would stay responsible.)

#4 Updated by mkittler over 1 year ago

According to my tests with a database dump from OSD using an extra JSON route would only improve the responsiveness of the page but the initial loading would still take some time.

The bottleneck is - according to nytprof - the database query and also the sorting/prioritizing of the results which is currently done in Perl code. Without sorting it takes only half of the time.

I can try to optimize that. Especially the sorting should be done by the database. However, this will be difficult because the criteria for the order is spread over multiple tables. But I'll give it a try.

Note that fast sorting would also be a prerequisite for making the table fully server-side because in this case the entire sorting must be done by the server for each user interaction with the table.

#5 Updated by mkittler over 1 year ago

  • Status changed from New to In Progress

PR: https://github.com/os-autoinst/openQA/pull/1750

The computation of the data is now faster and there are more tests. However, the table is still rendered as HTML elements and there's no caching. So the next step would be serving the data as JSON and implement caching. The cache would be refreshed when limit_assets runs or when the user passes some parameter to force the refresh.

Serving JSON via an additional route will be a bit tricky, though. This is because the 'Assets by group' list relies on the same data but can not easily be made up from some JSON data served by another route like it is possible with the DataTable.

#6 Updated by coolo over 1 year ago

  • Target version changed from Ready to Current Sprint
  • Difficulty changed from medium to hard

#7 Updated by mkittler over 1 year ago

PR: https://github.com/os-autoinst/openQA/pull/1821

The static/cache approach works basically. Locally it takes now 10 seconds to load (including client side rendering). The code and performance could still be improved, but at least the table is usable again.

A button to re-trigger the minion job is still missing. Right now it is only possible to use the query parameter force_refresh=1 but I suppose running the asset cleanup/status algorithm only via minion would be the better option.

It is still possible to delete assets within the table. However, the change is not immediately visible anymore due to the caching. Not sure how 'bad' this is (the cleanup is automated anyways).

#8 Updated by mkittler over 1 year ago

  • Status changed from In Progress to Feedback

Let's see how fast is loads in production. If it is at least usable again I would close the issue. Making it server-side would be a further enhancement.

#9 Updated by mkittler over 1 year ago

  • Status changed from Feedback to Resolved

It takes now < 25 seconds to load in production. Since the page isn't accessed that often I suppose that's ok (and it is already a huge improvement to how slow it was before).

So I'm closing the ticket. When further improvements are required (see comments on PR), just reopen or create a new ticket.

#10 Updated by coolo about 1 year ago

  • Target version changed from Current Sprint to Done

Also available in: Atom PDF