Project

General

Profile

Actions
Copy link

action #122776

closed

containers_tw_image_on_ubuntu_host: error 403 (forbidden) downloading the ubuntu qcow2 file

Added by dimstar almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
ph03nix
Category:
-
Target version:
QA (public) - future
Start date:
2023-01-06
Due date:
% Done:

100%

Estimated time:

Description

Seen in https://openqa.opensuse.org/tests/3017400 (multiple reruns; the day before this worked)

[2023-01-06T08:14:24.856791+01:00] [debug] Found HDD_1, caching ubuntu-22.04.qcow2
[2023-01-06T08:14:24.860952+01:00] [info] Downloading ubuntu-22.04.qcow2, request #490426 sent to Cache Service
[2023-01-06T08:14:35.043522+01:00] [info] Download of ubuntu-22.04.qcow2 processed:
[info] [#490426] Cache size of "/var/lib/openqa/cache" is 398 GiB, with limit 400 GiB
[info] [#490426] Downloading "ubuntu-22.04.qcow2" from "http://openqa1-opensuse/tests/3017400/asset/hdd/ubuntu-22.04.qcow2"
[info] [#490426] Download of "/var/lib/openqa/cache/openqa1-opensuse/ubuntu-22.04.qcow2" failed: 403 Forbidden

Actions
Copy link
 #1

Updated by mkittler almost 2 years ago

The URL is https://openqa.opensuse.org/assets/hdd/fixed/ubuntu-22.04.qcow2. The permissions look good in the file system but the apache2 error log shows the following:

[Tue Jan 10 12:52:39.287523 2023] [core:error] [pid 28619] [client 192.168.47.102:42126] AH00037: Symbolic link not allowed or link target not accessible: /var/lib/openqa/share/factory/hdd/fixed/ubuntu-22.04.qcow2

There were no recent changes in https://github.com/os-autoinst/openQA/tree/master/etc/apache2/vhosts.d.

Actions
Copy link
 #2

Updated by mkittler almost 2 years ago

Looks like symlinks are generally not allowed in our apache config. openQA was likely falling back the version of the file that was still present in the worker cache (although I currently fail to see the mistake in the error handling).

Actions
Copy link
 #3

Updated by ph03nix almost 2 years ago

I updated the images recently, so it's likely that I caused the issue thereby. I'm going to have a look.

Actions
Copy link
 #4

Updated by ph03nix almost 2 years ago

Can't assign myself, no idea why. But will still work on it.

Actions
Copy link
 #5

Updated by mkittler almost 2 years ago

I've been preparing a change to fix the error handling that made the investigation harder than it needed to be: https://github.com/os-autoinst/openQA/pull/4980

Note that I cannot assign myself as well. I suppose that's because this is a ticked within the openSUSE Release Process project. However, since @ph03nix already wants to work on the symlink issue itself and I only provided a PR for the cache service error handling it makes more sense when he's the assignee.

Actions
Copy link
 #6

Updated by dimstar almost 2 years ago

  • Project changed from openSUSE Release Process to openQA Infrastructure (public)
Actions
Copy link
 #7

Updated by dimstar almost 2 years ago

  • Assignee set to ph03nix
Actions
Copy link
 #8

Updated by okurz almost 2 years ago

  • Target version set to future
Actions
Copy link
 #9

Updated by ph03nix almost 2 years ago

For our apache config, we have the option +SymLinksIfOwnerMatch enabled. This means that symlinks are allowed, but only if the owner of the symlink is the same as the owner of the target file: https://github.com/os-autoinst/salt-states-openqa/blob/master/apache2/vhosts.d/openqa.conf#L15

However that applies only to files in /var/lib/openqa/share/factory.

Actions
Copy link
 #10

Updated by dimstar almost 2 years ago

ph03nix wrote:

For our apache config, we have the option +SymLinksIfOwnerMatch enabled. This means that symlinks are allowed, but only if the owner of the symlink is the same as the owner of the target file: https://github.com/os-autoinst/salt-states-openqa/blob/master/apache2/vhosts.d/openqa.conf#L15

That file does not match AT ALL what is present on Ariel

wc -l < openqa.conf
73

ls -la openqa.conf
-rw-r--r-- 1 root root 3354 Mar 8 2022 openqa.conf

vs a 19-line file in that salt state and a last-change-date from 2021

Actions
Copy link
 #11

Updated by ph03nix almost 2 years ago

The apache configs are confusing and do not match the salt state

For now, replaced the symlink with a hardlink:

ln ubuntu-22.04-20221229.qcow2 ubuntu-22.04.qcow2

Test run seems to be running again: https://openqa.opensuse.org/tests/3032265#details

Actions
Copy link
 #12

Updated by ph03nix almost 2 years ago

  • % Done changed from 0 to 90

Fixed it by adding a custom block to /etc/apache2/vhosts.d/openqa.conf:

    <Directory "/var/lib/openqa/share/factory/hdd">
        AllowOverride None
        Require all granted
        Options +Indexes +SymLinksIfOwnerMatch
    </Directory>

Restarted test run also works: https://openqa.opensuse.org/tests/3032318#live and the asset file can be downloaded.

Waiting for the test run to complete before closing the ticket.

Actions
Copy link
 #13

Updated by ph03nix almost 2 years ago

  • Status changed from New to Resolved
  • % Done changed from 90 to 100

Resolved.

Actions
Copy link
 #14

Updated by tinita almost 2 years ago

ph03nix wrote:

The apache configs are confusing and do not match the salt state

Yeah, we don't manage o3 with salt currently...

Actions
Copy link

Also available in: Atom PDF