Project

General

Profile

action #122776

containers_tw_image_on_ubuntu_host: error 403 (forbidden) downloading the ubuntu qcow2 file

Added by dimstar 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
ph03nix
Target version:
QA - future
Start date:
2023-01-06
Due date:
% Done:

100%

Estimated time:

Description

Seen in https://openqa.opensuse.org/tests/3017400 (multiple reruns; the day before this worked)

[2023-01-06T08:14:24.856791+01:00] [debug] Found HDD_1, caching ubuntu-22.04.qcow2
[2023-01-06T08:14:24.860952+01:00] [info] Downloading ubuntu-22.04.qcow2, request #490426 sent to Cache Service
[2023-01-06T08:14:35.043522+01:00] [info] Download of ubuntu-22.04.qcow2 processed:
[info] [#490426] Cache size of "/var/lib/openqa/cache" is 398 GiB, with limit 400 GiB
[info] [#490426] Downloading "ubuntu-22.04.qcow2" from "http://openqa1-opensuse/tests/3017400/asset/hdd/ubuntu-22.04.qcow2"
[info] [#490426] Download of "/var/lib/openqa/cache/openqa1-opensuse/ubuntu-22.04.qcow2" failed: 403 Forbidden

History

#1 Updated by mkittler 2 months ago

The URL is https://openqa.opensuse.org/assets/hdd/fixed/ubuntu-22.04.qcow2. The permissions look good in the file system but the apache2 error log shows the following:

[Tue Jan 10 12:52:39.287523 2023] [core:error] [pid 28619] [client 192.168.47.102:42126] AH00037: Symbolic link not allowed or link target not accessible: /var/lib/openqa/share/factory/hdd/fixed/ubuntu-22.04.qcow2

There were no recent changes in https://github.com/os-autoinst/openQA/tree/master/etc/apache2/vhosts.d.

#2 Updated by mkittler 2 months ago

Looks like symlinks are generally not allowed in our apache config. openQA was likely falling back the version of the file that was still present in the worker cache (although I currently fail to see the mistake in the error handling).

#3 Updated by ph03nix 2 months ago

I updated the images recently, so it's likely that I caused the issue thereby. I'm going to have a look.

#4 Updated by ph03nix 2 months ago

Can't assign myself, no idea why. But will still work on it.

#5 Updated by mkittler 2 months ago

I've been preparing a change to fix the error handling that made the investigation harder than it needed to be: https://github.com/os-autoinst/openQA/pull/4980

Note that I cannot assign myself as well. I suppose that's because this is a ticked within the openSUSE Release Process project. However, since ph03nix already wants to work on the symlink issue itself and I only provided a PR for the cache service error handling it makes more sense when he's the assignee.

#6 Updated by dimstar 2 months ago

  • Project changed from openSUSE Release Process to openQA Infrastructure

#7 Updated by dimstar 2 months ago

  • Assignee set to ph03nix

#8 Updated by okurz 2 months ago

  • Target version set to future

#9 Updated by ph03nix 2 months ago

For our apache config, we have the option +SymLinksIfOwnerMatch enabled. This means that symlinks are allowed, but only if the owner of the symlink is the same as the owner of the target file: https://github.com/os-autoinst/salt-states-openqa/blob/master/apache2/vhosts.d/openqa.conf#L15

However that applies only to files in /var/lib/openqa/share/factory.

#10 Updated by dimstar 2 months ago

ph03nix wrote:

For our apache config, we have the option +SymLinksIfOwnerMatch enabled. This means that symlinks are allowed, but only if the owner of the symlink is the same as the owner of the target file: https://github.com/os-autoinst/salt-states-openqa/blob/master/apache2/vhosts.d/openqa.conf#L15

That file does not match AT ALL what is present on Ariel

wc -l < openqa.conf
73

ls -la openqa.conf
-rw-r--r-- 1 root root 3354 Mar 8 2022 openqa.conf

vs a 19-line file in that salt state and a last-change-date from 2021

#11 Updated by ph03nix 2 months ago

The apache configs are confusing and do not match the salt state

For now, replaced the symlink with a hardlink:

ln ubuntu-22.04-20221229.qcow2 ubuntu-22.04.qcow2

Test run seems to be running again: https://openqa.opensuse.org/tests/3032265#details

#12 Updated by ph03nix 2 months ago

  • % Done changed from 0 to 90

Fixed it by adding a custom block to /etc/apache2/vhosts.d/openqa.conf:

    <Directory "/var/lib/openqa/share/factory/hdd">
        AllowOverride None
        Require all granted
        Options +Indexes +SymLinksIfOwnerMatch
    </Directory>

Restarted test run also works: https://openqa.opensuse.org/tests/3032318#live and the asset file can be downloaded.

Waiting for the test run to complete before closing the ticket.

#13 Updated by ph03nix 2 months ago

  • Status changed from New to Resolved
  • % Done changed from 90 to 100

Resolved.

#14 Updated by tinita 2 months ago

ph03nix wrote:

The apache configs are confusing and do not match the salt state

Yeah, we don't manage o3 with salt currently...

Also available in: Atom PDF