Project

General

Profile

action #69637

Faster localhost uploads from worker to webui

Added by kraih 3 months ago. Updated 14 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-08-06
Due date:
% Done:

0%

Estimated time:
Difficulty:
hard

Description

Motivation

The chunked asset uploader we use to move assets from workers to the webui is extremely inefficient. This is especially noticeable when worker and webui are running on the same host and very large assets have to be uploaded. This has been a big problem with SUSE Manager for Retail tests, to the point where an optimization could save multiple hours per day.
A similar problem has been reported by "marmarek" in [#opensuse-factory](irc://chat.freenode.net/opensuse-factory)

Acceptance criteria

  • AC1: Asset upload on "localhost" (worker and webui on same machine) is significantly faster, e.g. ask marmarek for verification in his environment
  • AC2: Asset upload on non-localhost workers still use remote upload methods with no significant performance degradation

Suggestions

  • Try to add a "secondary code path" to just move files between local folders (regardless if same or differing filesystem) on the same machine instead of uploading them completely via HTTP

History

#1 Updated by kraih 3 months ago

  • Assignee set to kraih
  • Target version set to Ready
  • Difficulty set to hard

I have looked a bit at the code and i think we can add a secondary code path to move files on the file system instead of actually uploading them via HTTP.

#2 Updated by okurz 3 months ago

  • Tags set to upload, worker

#3 Updated by okurz 3 months ago

  • Subject changed from Slow localhost uploads from worker to webui to Faster localhost uploads from worker to webui
  • Description updated (diff)
  • Status changed from New to Workable

hi, thanks for the ticket. I made it "Workable" and focussed on what you wrote in the subject about "localhost uploads", not a generic performance improvement which we might have as a side-effect but it should not be the focus here. Any necessary refactoring to have the code in a clean state and better testable is of course included.

#4 Updated by cdywan about 2 months ago

  • Status changed from Workable to Feedback

#5 Updated by mkittler about 2 months ago

  • Status changed from Feedback to Resolved

We've also already got feedback within the PR comments so I'm marking the issue as resolved. Waiting whether it works on the main production instances is not useful for this feature anyways.

#6 Updated by okurz about 2 months ago

  • Status changed from Resolved to Feedback

but the PR nicely explains that the config option could be turned into a new default. As there was good success reported but no problems I suggest we change to local upload by default with the option to disable which we might even want to use in the full stack tests.

#7 Updated by kraih 21 days ago

Agreed, reversing the option and enabling the feature by default for localhost uploads does make sense. All feedback i got so far has been very positive.

#8 Updated by okurz 21 days ago

  • Status changed from Feedback to Workable

alright, plan to do that? If not, please create a new ticket for that and set the status here back to "Resolved".

#9 Updated by kraih 17 days ago

  • Status changed from Workable to In Progress

Sure, i don't mind doing it.

#10 Updated by kraih 14 days ago

#11 Updated by okurz 14 days ago

  • Status changed from In Progress to Resolved

PR merged. As the feature was confirmed working by others already in the past I regard this as "Resolved". I assume you haven't plan for other tasks in this ticket. Thanks for your work.

Also available in: Atom PDF