action #122776
closed
containers_tw_image_on_ubuntu_host: error 403 (forbidden) downloading the ubuntu qcow2 file
Added by dimstar almost 2 years ago.
Updated almost 2 years ago.
Description
Seen in https://openqa.opensuse.org/tests/3017400 (multiple reruns; the day before this worked)
[2023-01-06T08:14:24.856791+01:00] [debug] Found HDD_1, caching ubuntu-22.04.qcow2
[2023-01-06T08:14:24.860952+01:00] [info] Downloading ubuntu-22.04.qcow2, request #490426 sent to Cache Service
[2023-01-06T08:14:35.043522+01:00] [info] Download of ubuntu-22.04.qcow2 processed:
[info] [#490426] Cache size of "/var/lib/openqa/cache" is 398 GiB, with limit 400 GiB
[info] [#490426] Downloading "ubuntu-22.04.qcow2" from "http://openqa1-opensuse/tests/3017400/asset/hdd/ubuntu-22.04.qcow2"
[info] [#490426] Download of "/var/lib/openqa/cache/openqa1-opensuse/ubuntu-22.04.qcow2" failed: 403 Forbidden
Looks like symlinks are generally not allowed in our apache config. openQA was likely falling back the version of the file that was still present in the worker cache (although I currently fail to see the mistake in the error handling).
I updated the images recently, so it's likely that I caused the issue thereby. I'm going to have a look.
Can't assign myself, no idea why. But will still work on it.
I've been preparing a change to fix the error handling that made the investigation harder than it needed to be: https://github.com/os-autoinst/openQA/pull/4980
Note that I cannot assign myself as well. I suppose that's because this is a ticked within the openSUSE Release Process project. However, since @ph03nix already wants to work on the symlink issue itself and I only provided a PR for the cache service error handling it makes more sense when he's the assignee.
- Project changed from openSUSE Release Process to openQA Infrastructure (public)
- Target version set to future
ph03nix wrote:
For our apache config, we have the option +SymLinksIfOwnerMatch enabled. This means that symlinks are allowed, but only if the owner of the symlink is the same as the owner of the target file: https://github.com/os-autoinst/salt-states-openqa/blob/master/apache2/vhosts.d/openqa.conf#L15
That file does not match AT ALL what is present on Ariel
wc -l < openqa.conf
73
ls -la openqa.conf
-rw-r--r-- 1 root root 3354 Mar 8 2022 openqa.conf
vs a 19-line file in that salt state and a last-change-date from 2021
- % Done changed from 0 to 90
Fixed it by adding a custom block to /etc/apache2/vhosts.d/openqa.conf:
<Directory "/var/lib/openqa/share/factory/hdd">
AllowOverride None
Require all granted
Options +Indexes +SymLinksIfOwnerMatch
</Directory>
Restarted test run also works: https://openqa.opensuse.org/tests/3032318#live and the asset file can be downloaded.
Waiting for the test run to complete before closing the ticket.
- Status changed from New to Resolved
- % Done changed from 90 to 100
ph03nix wrote:
The apache configs are confusing and do not match the salt state
Yeah, we don't manage o3 with salt currently...
Also available in: Atom
PDF