action #54827

[cache] downloading assets with `:` or `#` in filenames using the caching fails

Added by okurz 7 months ago. Updated about 1 month ago.

Status:ResolvedStart date:30/07/2019
Priority:NormalDue date:
Assignee:kraih% Done:

0%

Category:Feature requests
Target version:Done
Difficulty:
Duration:

Description

Motivation

See #52499#note-21

Steps to reproduce

  • In a setup with a caching worker
  • Prepare a downloadable asset with : or # in the filename
  • Trigger/clone a job using the asset in e.g. ISO or HDD parameter
  • Observe the problem in the worker log

Expected result

: or # should be valid characters, also as they work within openQA as an asset when caching is not involved.

Workaround

Avoid : and # in the filenames of assets or avoid caching on the worker


Related issues

Copied from openQA Project - action #54809: PUBLISH_HDD_1 fails with '/' in variable name Resolved 30/07/2019

History

#1 Updated by okurz 7 months ago

  • Copied from action #54809: PUBLISH_HDD_1 fails with '/' in variable name added

#2 Updated by okurz 4 months ago

  • Description updated (diff)

#3 Updated by Xiaojing_liu about 1 month ago

I did some tests on this ticket, the downloading assets with : did not fail. But the assets with # failed. Because # is a special character that needs to be encoded to %23 in URL.

Here is my test result:

[2020-01-03T16:37:43.0911 CST] [info] Download of c:.iso processed:
[info] [#15] Cache size of "/var/lib/openqa/cache" is 372MiB, with limit 50GiB
[info] [#15] Downloading "c:.iso" from "http://10.67.19.103/tests/15/asset/iso/c:.iso"
[info] [#15] Size of "/var/lib/openqa/cache/10.67.19.103/c:.iso" is 6MiB, with ETag ""600000-59b3837bdf695""
[info] [#15] Download of "/var/lib/openqa/cache/10.67.19.103/c:.iso" successful, new cache size is 378MiB

The asset with : was downloaded successfully.

So I would like to confirm that if we need to fix this issue about #?

#4 Updated by okurz about 1 month ago

well, I think it is fine if you just ensure necessary characters in filesnames are encoded in corresponding URLs. If you did not find a problem with ":" I recommend to still cover it in a test as well.

#5 Updated by okurz about 1 month ago

  • Status changed from New to Workable

#6 Updated by kraih about 1 month ago

It shouldn't be too hard to make # work for downloads too if we construct the URL a little differently in the cache service. Making a test for it will be much harder. :)

#7 Updated by Xiaojing_liu about 1 month ago

  • Status changed from Workable to In Progress
  • Assignee set to Xiaojing_liu

#9 Updated by kraih about 1 month ago

  • Status changed from In Progress to Feedback

#10 Updated by okurz about 1 month ago

  • Assignee changed from Xiaojing_liu to kraih
  • Priority changed from Low to Urgent
  • Target version set to Current Sprint

trouble caused https://openqa.suse.de/tests/3764075 , reverted osd workers with

sudo salt -l error -C 'G@roles:worker' cmd.run 'sudo zypper -n in -f --oldpackage /var/cache/zypp/packages/devel_openQA/noarch/openQA-{common,client,worker}-4.6.1577344382.5aaa6ced5-lp151.2128.1.noarch.rpm'

side-issue about the UX: "another thing is the feedback. it's not really obvious from the jobs what's wrong". "it is if you look closely, the download target is just a path without host :). but yea, not sure why it doesn't fail more spectacularly"

#11 Updated by kraih about 1 month ago

  • Priority changed from Urgent to High

The main problem with the previous fix should be resolved again. https://github.com/os-autoinst/openQA/pull/2645

#12 Updated by kraih about 1 month ago

  • Priority changed from High to Normal

#13 Updated by okurz about 1 month ago

  • Status changed from Feedback to Resolved
  • Target version changed from Current Sprint to Done

State was deployed again with fix on OSD yesterday, no fallout. I guess we can resolve it now. I will just resolve it, you can reopen it if I misunderstood and you still plan some steps.

Also available in: Atom PDF