action #54827
closed[cache] downloading assets with `:` or `#` in filenames using the caching fails
Description
Motivation¶
See #52499#note-21
Steps to reproduce¶
- In a setup with a caching worker
- Prepare a downloadable asset with
:or#in the filename - Trigger/clone a job using the asset in e.g. ISO or HDD parameter
- Observe the problem in the worker log
Expected result¶
: or # should be valid characters, also as they work within openQA as an asset when caching is not involved.
Workaround¶
Avoid : and # in the filenames of assets or avoid caching on the worker
Updated by okurz over 5 years ago
- Copied from action #54809: PUBLISH_HDD_1 fails with '/' in variable name added
Updated by Xiaojing_liu almost 5 years ago
I did some tests on this ticket, the downloading assets with : did not fail. But the assets with # failed. Because # is a special character that needs to be encoded to %23 in URL.
Here is my test result:
[2020-01-03T16:37:43.0911 CST] [info] Download of c:.iso processed:
[info] [#15] Cache size of "/var/lib/openqa/cache" is 372MiB, with limit 50GiB
[info] [#15] Downloading "c:.iso" from "http://10.67.19.103/tests/15/asset/iso/c:.iso"
[info] [#15] Size of "/var/lib/openqa/cache/10.67.19.103/c:.iso" is 6MiB, with ETag ""600000-59b3837bdf695""
[info] [#15] Download of "/var/lib/openqa/cache/10.67.19.103/c:.iso" successful, new cache size is 378MiB
The asset with : was downloaded successfully.
So I would like to confirm that if we need to fix this issue about #?
Updated by okurz almost 5 years ago
well, I think it is fine if you just ensure necessary characters in filesnames are encoded in corresponding URLs. If you did not find a problem with ":" I recommend to still cover it in a test as well.
Updated by kraih almost 5 years ago
It shouldn't be too hard to make # work for downloads too if we construct the URL a little differently in the cache service. Making a test for it will be much harder. :)
Updated by Xiaojing_liu almost 5 years ago
- Status changed from Workable to In Progress
- Assignee set to Xiaojing_liu
Updated by Xiaojing_liu almost 5 years ago
Updated by kraih almost 5 years ago
- Status changed from In Progress to Feedback
This should be resolved now. https://github.com/os-autoinst/openQA/pull/2639
Updated by okurz almost 5 years ago
- Assignee changed from Xiaojing_liu to kraih
- Priority changed from Low to Urgent
- Target version set to Current Sprint
trouble caused https://openqa.suse.de/tests/3764075 , reverted osd workers with
sudo salt -l error -C 'G@roles:worker' cmd.run 'sudo zypper -n in -f --oldpackage /var/cache/zypp/packages/devel_openQA/noarch/openQA-{common,client,worker}-4.6.1577344382.5aaa6ced5-lp151.2128.1.noarch.rpm'
side-issue about the UX: "another thing is the feedback. it's not really obvious from the jobs what's wrong". "it is if you look closely, the download target is just a path without host :). but yea, not sure why it doesn't fail more spectacularly"
Updated by kraih almost 5 years ago
- Priority changed from Urgent to High
The main problem with the previous fix should be resolved again. https://github.com/os-autoinst/openQA/pull/2645
Updated by okurz almost 5 years ago
- Status changed from Feedback to Resolved
- Target version changed from Current Sprint to Done
State was deployed again with fix on OSD yesterday, no fallout. I guess we can resolve it now. I will just resolve it, you can reopen it if I misunderstood and you still plan some steps.