Project

General

Profile

Actions

action #160171

closed

[openQA][assets] Access to openQA assets forbidden size:S

Added by waynechen55 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2024-05-10
Due date:
2024-05-27
% Done:

0%

Estimated time:

Description

Observation

Browse assets on openqa.suse.de is forbidden. Can not see any medium regardless of medium types, including repo, hdd, iso or fixed.

Steps to reproduce

Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
call openqa-query-for-job-label poo#160171

Impact

Can not determine what is really stored out there on openqa.suse.de

Problem

May have something to do with web server.

Suggestions

  • Check web server settings
  • Check services running

Workaround

n/a


Files

osd_assets_forbidden.png (41 KB) osd_assets_forbidden.png waynechen55, 2024-05-10 05:37

Related issues 1 (0 open1 closed)

Related to openQA Project - action #130636: high response times on osd - Try nginx on OSD size:SResolvedmkittler2024-05-17

Actions
Actions #1

Updated by waynechen55 3 months ago

Actions #2

Updated by okurz 3 months ago

  • Category set to Regressions/Crashes
  • Target version set to Ready

Is this a regression or has that never worked in before?

Actions #3

Updated by waynechen55 3 months ago

okurz wrote in #note-2:

Is this a regression or has that never worked in before?

Regression. This works very well before.

Actions #4

Updated by waynechen55 3 months ago

  • File deleted (vm_power_off.png)
Actions #5

Updated by Julie_CAO 2 months ago

In some cases openqa tests failed in accessing this repo: https://10.145.10.207/tests/14278456#step/installation/4

Actions #6

Updated by waynechen55 2 months ago

This issue causes openQA test run failure https://openqa.suse.de/tests/14286723 @okurz

Actions #7

Updated by xlai 2 months ago

  • Priority changed from Normal to Urgent
Actions #8

Updated by nanzhang 2 months ago · Edited

The same issue, it caused vmware online upgrade test failure - https://openqa.suse.de/tests/14279862

Actions #9

Updated by waynechen55 2 months ago

waynechen55 wrote in #note-6:

This issue causes openQA test run failure https://openqa.suse.de/tests/14286723 @okurz

I think this failure is caused by incorrect setting of test suite which was triggered mistakenly somehow. But anyway I think this issue should be fixed.

Actions #10

Updated by ybonatakis 2 months ago · Edited

If this is used to work before, the easiest mitigation could be to switch to apache. However this endpoint doesnt seem to work on my local instance even with apache

Actions #11

Updated by okurz 2 months ago

  • Related to action #130636: high response times on osd - Try nginx on OSD size:S added
Actions #12

Updated by mkittler 2 months ago

  • Status changed from New to In Progress
  • Assignee set to mkittler
Actions #14

Updated by okurz 2 months ago · Edited

  • Subject changed from [openQA][assets] Access to openQA assets forbidden to [openQA][assets] Access to openQA assets forbidden auto_review:"Download (curl) error for 'http://openqa.suse.de/":retry

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1181 merged and deployed.

And called

export host=openqa.suse.de; failed_since="'2024-05-08'" ./openqa-monitor-investigation-candidates | ./openqa-label-known-issues-multi
Actions #16

Updated by okurz 2 months ago

  • Description updated (diff)
Actions #17

Updated by okurz 2 months ago

  • Subject changed from [openQA][assets] Access to openQA assets forbidden auto_review:"Download (curl) error for 'http://openqa.suse.de/":retry to [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry
Actions #18

Updated by okurz 2 months ago · Edited

Seems like no further jobs were labeled. Rerunning with generalized regex to get rid of potentially problematic '()' and the "'"

Plus

export host=openqa.suse.de; export failed_since="'2024-05-08'"; for i in $(./openqa-monitor-investigation-candidates | sed 's@^[^0-9]*\([0-9]*\) .*$@\1@g'); do openqa-cli api --osd -X post jobs/$i/duplicate ; done

And for jobs that now are labeled with the current ticket openqa-query-for-job-label poo#160171 yields:

14288091|2024-05-13 12:12:04|done|failed|select_modules_and_patterns+registration||grenache-1
14288093|2024-05-13 11:39:27|done|failed|ha_textmode_skip_registration||grenache-1
14287663|2024-05-13 07:33:20|done|incomplete|online_upgrade_sles15sp5_vmware|abandoned: associated worker sapworker2:7 re-connected but abandoned the job|sapworker2
14279413|2024-05-13 07:15:07|done|failed|sev-es-gi-guest_sles15sp5-on-host_developing-kvm||imagetester
14279434|2024-05-12 10:50:16|done|failed|prepare-host_developing-xen||worker35
14281144|2024-05-11 18:36:10|done|failed|SAPHanaSR_ScaleUp_PerfOpt_WMP_node01||worker35
14279360|2024-05-11 17:37:53|done|failed|create_hdd_textmode||worker33
14281003|2024-05-11 17:21:27|done|failed|fips_env_mode_tests_crypt_tool||worker31
14281009|2024-05-11 17:20:21|done|failed|fips_ker_mode_tests_crypt_tool||worker33
14281246|2024-05-11 16:56:09|done|failed|jeos-ltp-commands||worker34
Actions #19

Updated by okurz 2 months ago

  • Due date set to 2024-05-27
  • Status changed from In Progress to Feedback
  • Priority changed from Urgent to Normal

this should reduce urgency. I assume you don't plan further implementation so we can monitor. I suggest to review the scenarios linked to the labeled failed jobs before resolving so that this ticket wouldn't be reopened afterwards.

Actions #20

Updated by mkittler 2 months ago

sounds good

Actions #21

Updated by mkittler 2 months ago · Edited

I checked some scenarios. Two scenarios are still failing:

Maybe #160239 was still causing these tests to fail (considering https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1188 was merged only 24 minutes ago).

Right now I can download e.g. https://openqa.suse.de/assets/repo/SLE-15-SP6-Full-ppc64le-Build88.1-Media1/Module-Basesystem/ppc64le/keyutils-1.6.3-5.6.1.ppc64le.rpm just fine and listing e.g. https://openqa.suse.de/assets/repo/SLE-15-SP6-Full-ppc64le-Build88.1-Media1 works as well. (So not sure why these tests fails due to /var/log/YaST2/y2log-1.gz:2024-05-13 17:28:13 <5> grenache-7(3908) [zypp-core] Exception.cc(log):202 MediaCurl.cc(getFileCopy):795 RETHROW: Downloaded data exceeded the expected filesize '117.6 KiB' of 'http://openqa.suse.de/assets/repo/SLE-15-SP6-Full-ppc64le-Build88.1-Media1/Module-Basesystem/ppc64le/keyutils-1.6.3-5.6.1.ppc64le.rpm'..)


The majority of scenarios look good now:


Some scenarios don't seem to be relevant:

Actions #22

Updated by okurz 2 months ago

mkittler wrote in #note-21:

I checked some scenarios. Two scenarios are still failing:

[…]

Maybe #160239 was still causing these tests to fail (considering https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1188 was merged only 24 minutes ago).

yes, my mistake.

Actions #23

Updated by waynechen55 2 months ago · Edited

I can access to assets folder on openqa.suse.de now.

Actions #24

Updated by livdywan 2 months ago

  • Subject changed from [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry to [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry size:s
Actions #25

Updated by okurz 2 months ago

  • Subject changed from [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry size:s to [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry size:S
Actions #26

Updated by okurz 2 months ago

$ openqa-query-for-job-label poo#160171
14337364|2024-05-17 07:49:45|done|parallel_failed|cc_net_case_client||worker29
14337335|2024-05-17 07:38:38|done|failed|cc_net_case_server||worker-arm1
14336776|2024-05-17 06:21:38|done|parallel_failed|cc_net_case_client_uefi||worker29
14336775|2024-05-17 06:21:37|done|failed|cc_net_case_server_uefi||worker34
14336773|2024-05-17 06:19:41|done|failed|cc_net_case_server||worker31
14336778|2024-05-17 06:18:35|done|failed|cc_net_case_client||worker-arm1
14336781|2024-05-17 06:17:32|done|failed|cc_audit_remote_server||worker-arm2
14336767|2024-05-17 06:14:44|done|failed|cc_audit_remote_client_uefi||worker40
14336766|2024-05-17 06:14:44|done|failed|cc_audit_remote_server_uefi||worker31
14336764|2024-05-17 06:14:35|done|failed|cc_audit_remote_server||worker30
Actions #27

Updated by mkittler 2 months ago

  • Subject changed from [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry size:S to [openQA][assets] Access to openQA assets forbidden size:S
  • Status changed from Feedback to Resolved

None of those jobs are relevant. They failed due to:

Error message: Could not resolve host: openqa.suse.de

That error has a completely different cause. Considering I fixed the serving of assets we can be sure that this issue is no longer the cause of download errors so I'm removing the auto review regex.

At this point too many jobs have been mislabeled to remove labels of relevant jobs manually. So I just ran delete from comments where text like '%poo#160171%'; to delete all comments referencing this issue.

With that I would also consider this ticket resolved. (The scenarios mentioned in #160171#note-21 look still good.)

Actions

Also available in: Atom PDF