Project

General

Profile

Actions

action #54827

closed

[cache] downloading assets with `:` or `#` in filenames using the caching fails

Added by okurz over 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2019-07-30
Due date:
% Done:

0%

Estimated time:

Description

Motivation

See #52499#note-21

Steps to reproduce

  • In a setup with a caching worker
  • Prepare a downloadable asset with : or # in the filename
  • Trigger/clone a job using the asset in e.g. ISO or HDD parameter
  • Observe the problem in the worker log

Expected result

: or # should be valid characters, also as they work within openQA as an asset when caching is not involved.

Workaround

Avoid : and # in the filenames of assets or avoid caching on the worker


Related issues 1 (0 open1 closed)

Copied from openQA Project - action #54809: PUBLISH_HDD_1 fails with '/' in variable nameResolvedXiaojing_liu2019-07-30

Actions
Actions #1

Updated by okurz over 5 years ago

  • Copied from action #54809: PUBLISH_HDD_1 fails with '/' in variable name added
Actions #2

Updated by okurz about 5 years ago

  • Description updated (diff)
Actions #3

Updated by Xiaojing_liu almost 5 years ago

I did some tests on this ticket, the downloading assets with : did not fail. But the assets with # failed. Because # is a special character that needs to be encoded to %23 in URL.

Here is my test result:

[2020-01-03T16:37:43.0911 CST] [info] Download of c:.iso processed:
[info] [#15] Cache size of "/var/lib/openqa/cache" is 372MiB, with limit 50GiB
[info] [#15] Downloading "c:.iso" from "http://10.67.19.103/tests/15/asset/iso/c:.iso"
[info] [#15] Size of "/var/lib/openqa/cache/10.67.19.103/c:.iso" is 6MiB, with ETag ""600000-59b3837bdf695""
[info] [#15] Download of "/var/lib/openqa/cache/10.67.19.103/c:.iso" successful, new cache size is 378MiB

The asset with : was downloaded successfully.

So I would like to confirm that if we need to fix this issue about #?

Actions #4

Updated by okurz almost 5 years ago

well, I think it is fine if you just ensure necessary characters in filesnames are encoded in corresponding URLs. If you did not find a problem with ":" I recommend to still cover it in a test as well.

Actions #5

Updated by okurz almost 5 years ago

  • Status changed from New to Workable
Actions #6

Updated by kraih almost 5 years ago

It shouldn't be too hard to make # work for downloads too if we construct the URL a little differently in the cache service. Making a test for it will be much harder. :)

Actions #7

Updated by Xiaojing_liu almost 5 years ago

  • Status changed from Workable to In Progress
  • Assignee set to Xiaojing_liu
Actions #9

Updated by kraih almost 5 years ago

  • Status changed from In Progress to Feedback
Actions #10

Updated by okurz almost 5 years ago

  • Assignee changed from Xiaojing_liu to kraih
  • Priority changed from Low to Urgent
  • Target version set to Current Sprint

trouble caused https://openqa.suse.de/tests/3764075 , reverted osd workers with

sudo salt -l error -C 'G@roles:worker' cmd.run 'sudo zypper -n in -f --oldpackage /var/cache/zypp/packages/devel_openQA/noarch/openQA-{common,client,worker}-4.6.1577344382.5aaa6ced5-lp151.2128.1.noarch.rpm'

side-issue about the UX: "another thing is the feedback. it's not really obvious from the jobs what's wrong". "it is if you look closely, the download target is just a path without host :). but yea, not sure why it doesn't fail more spectacularly"

Actions #11

Updated by kraih almost 5 years ago

  • Priority changed from Urgent to High

The main problem with the previous fix should be resolved again. https://github.com/os-autoinst/openQA/pull/2645

Actions #12

Updated by kraih almost 5 years ago

  • Priority changed from High to Normal
Actions #13

Updated by okurz almost 5 years ago

  • Status changed from Feedback to Resolved
  • Target version changed from Current Sprint to Done

State was deployed again with fix on OSD yesterday, no fallout. I guess we can resolve it now. I will just resolve it, you can reopen it if I misunderstood and you still plan some steps.

Actions

Also available in: Atom PDF