Project

General

Profile

Actions
Copy link

action #69637

closed

Faster localhost uploads from worker to webui

Added by kraih over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
kraih
Category:
Feature requests
Target version:
Ready
Start date:
2020-08-06
Due date:
% Done:

0%

Estimated time:
Tags:
upload, worker

Description

Motivation

The chunked asset uploader we use to move assets from workers to the webui is extremely inefficient. This is especially noticeable when worker and webui are running on the same host and very large assets have to be uploaded. This has been a big problem with SUSE Manager for Retail tests, to the point where an optimization could save multiple hours per day.
A similar problem has been reported by "marmarek" in #opensuse-factory

Acceptance criteria

  • AC1: Asset upload on "localhost" (worker and webui on same machine) is significantly faster, e.g. ask marmarek for verification in his environment
  • AC2: Asset upload on non-localhost workers still use remote upload methods with no significant performance degradation

Suggestions

  • Try to add a "secondary code path" to just move files between local folders (regardless if same or differing filesystem) on the same machine instead of uploading them completely via HTTP
Actions
Copy link
 #1

Updated by kraih over 4 years ago

  • Assignee set to kraih
  • Target version set to Ready
  • Difficulty set to hard

I have looked a bit at the code and i think we can add a secondary code path to move files on the file system instead of actually uploading them via HTTP.

Actions
Copy link
 #2

Updated by okurz over 4 years ago

  • Tags set to upload, worker
Actions
Copy link
 #3

Updated by okurz over 4 years ago

  • Subject changed from Slow localhost uploads from worker to webui to Faster localhost uploads from worker to webui
  • Description updated (diff)
  • Status changed from New to Workable

hi, thanks for the ticket. I made it "Workable" and focussed on what you wrote in the subject about "localhost uploads", not a generic performance improvement which we might have as a side-effect but it should not be the focus here. Any necessary refactoring to have the code in a clean state and better testable is of course included.

Actions
Copy link
 #4

Updated by livdywan over 4 years ago

  • Status changed from Workable to Feedback
Actions
Copy link
 #5

Updated by mkittler over 4 years ago

  • Status changed from Feedback to Resolved

We've also already got feedback within the PR comments so I'm marking the issue as resolved. Waiting whether it works on the main production instances is not useful for this feature anyways.

Actions
Copy link
 #6

Updated by okurz over 4 years ago

  • Status changed from Resolved to Feedback

but the PR nicely explains that the config option could be turned into a new default. As there was good success reported but no problems I suggest we change to local upload by default with the option to disable which we might even want to use in the full stack tests.

Actions
Copy link
 #7

Updated by kraih about 4 years ago

Agreed, reversing the option and enabling the feature by default for localhost uploads does make sense. All feedback i got so far has been very positive.

Actions
Copy link
 #8

Updated by okurz about 4 years ago

  • Status changed from Feedback to Workable

alright, plan to do that? If not, please create a new ticket for that and set the status here back to "Resolved".

Actions
Copy link
 #9

Updated by kraih about 4 years ago

  • Status changed from Workable to In Progress

Sure, i don't mind doing it.

Actions
Copy link
 #10

Updated by kraih about 4 years ago

Actions
Copy link
 #11

Updated by okurz about 4 years ago

  • Status changed from In Progress to Resolved

PR merged. As the feature was confirmed working by others already in the past I regard this as "Resolved". I assume you haven't plan for other tasks in this ticket. Thanks for your work.

Actions
Copy link

Also available in: Atom PDF