action #69637
closedFaster localhost uploads from worker to webui
0%
Description
Motivation¶
The chunked asset uploader we use to move assets from workers to the webui is extremely inefficient. This is especially noticeable when worker and webui are running on the same host and very large assets have to be uploaded. This has been a big problem with SUSE Manager for Retail tests, to the point where an optimization could save multiple hours per day.
A similar problem has been reported by "marmarek" in #opensuse-factory
Acceptance criteria¶
- AC1: Asset upload on "localhost" (worker and webui on same machine) is significantly faster, e.g. ask marmarek for verification in his environment
- AC2: Asset upload on non-localhost workers still use remote upload methods with no significant performance degradation
Suggestions¶
- Try to add a "secondary code path" to just move files between local folders (regardless if same or differing filesystem) on the same machine instead of uploading them completely via HTTP
Updated by kraih over 3 years ago
- Assignee set to kraih
- Target version set to Ready
- Difficulty set to hard
I have looked a bit at the code and i think we can add a secondary code path to move files on the file system instead of actually uploading them via HTTP.
Updated by okurz over 3 years ago
- Subject changed from Slow localhost uploads from worker to webui to Faster localhost uploads from worker to webui
- Description updated (diff)
- Status changed from New to Workable
hi, thanks for the ticket. I made it "Workable" and focussed on what you wrote in the subject about "localhost uploads", not a generic performance improvement which we might have as a side-effect but it should not be the focus here. Any necessary refactoring to have the code in a clean state and better testable is of course included.
Updated by livdywan over 3 years ago
- Status changed from Workable to Feedback
The PR was merged: https://github.com/os-autoinst/openQA/pull/3325
Updated by mkittler over 3 years ago
- Status changed from Feedback to Resolved
We've also already got feedback within the PR comments so I'm marking the issue as resolved. Waiting whether it works on the main production instances is not useful for this feature anyways.
Updated by okurz over 3 years ago
- Status changed from Resolved to Feedback
but the PR nicely explains that the config option could be turned into a new default. As there was good success reported but no problems I suggest we change to local upload by default with the option to disable which we might even want to use in the full stack tests.
Updated by kraih over 3 years ago
Agreed, reversing the option and enabling the feature by default for localhost uploads does make sense. All feedback i got so far has been very positive.
Updated by okurz over 3 years ago
- Status changed from Feedback to Workable
alright, plan to do that? If not, please create a new ticket for that and set the status here back to "Resolved".
Updated by kraih over 3 years ago
- Status changed from Workable to In Progress
Sure, i don't mind doing it.
Updated by kraih over 3 years ago
Pull request has been opened. https://github.com/os-autoinst/openQA/pull/3458
Updated by okurz over 3 years ago
- Status changed from In Progress to Resolved
PR merged. As the feature was confirmed working by others already in the past I regard this as "Resolved". I assume you haven't plan for other tasks in this ticket. Thanks for your work.