Faster localhost uploads from worker to webui
The chunked asset uploader we use to move assets from workers to the webui is extremely inefficient. This is especially noticeable when worker and webui are running on the same host and very large assets have to be uploaded. This has been a big problem with SUSE Manager for Retail tests, to the point where an optimization could save multiple hours per day.
A similar problem has been reported by "marmarek" in [#opensuse-factory](irc://chat.freenode.net/opensuse-factory)
- AC1: Asset upload on "localhost" (worker and webui on same machine) is significantly faster, e.g. ask marmarek for verification in his environment
- AC2: Asset upload on non-localhost workers still use remote upload methods with no significant performance degradation
- Try to add a "secondary code path" to just move files between local folders (regardless if same or differing filesystem) on the same machine instead of uploading them completely via HTTP
- Subject changed from Slow localhost uploads from worker to webui to Faster localhost uploads from worker to webui
- Description updated (diff)
- Status changed from New to Workable
hi, thanks for the ticket. I made it "Workable" and focussed on what you wrote in the subject about "localhost uploads", not a generic performance improvement which we might have as a side-effect but it should not be the focus here. Any necessary refactoring to have the code in a clean state and better testable is of course included.
- Status changed from Resolved to Feedback
but the PR nicely explains that the config option could be turned into a new default. As there was good success reported but no problems I suggest we change to local upload by default with the option to disable which we might even want to use in the full stack tests.