action #160171
closed[openQA][assets] Access to openQA assets forbidden size:S
Added by waynechen55 about 2 months ago. Updated about 1 month ago.
0%
Description
Observation¶
Browse assets on openqa.suse.de is forbidden. Can not see any medium regardless of medium types, including repo, hdd, iso or fixed.
Steps to reproduce¶
- Open webpage like https://openqa.suse.de/assets/repo/
Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
call openqa-query-for-job-label poo#160171
Impact¶
Can not determine what is really stored out there on openqa.suse.de
Problem¶
May have something to do with web server.
Suggestions¶
- Check web server settings
- Check services running
Workaround¶
n/a
Files
Updated by waynechen55 about 2 months ago
- File osd_assets_forbidden.png osd_assets_forbidden.png added
- Description updated (diff)
Updated by okurz about 2 months ago
- Category set to Regressions/Crashes
- Target version set to Ready
Is this a regression or has that never worked in before?
Updated by waynechen55 about 2 months ago
okurz wrote in #note-2:
Is this a regression or has that never worked in before?
Regression. This works very well before.
Updated by Julie_CAO about 2 months ago
In some cases openqa tests failed in accessing this repo: https://10.145.10.207/tests/14278456#step/installation/4
Updated by waynechen55 about 2 months ago
This issue causes openQA test run failure https://openqa.suse.de/tests/14286723 @okurz
Updated by nanzhang about 2 months ago · Edited
The same issue, it caused vmware online upgrade test failure - https://openqa.suse.de/tests/14279862
Updated by waynechen55 about 2 months ago
waynechen55 wrote in #note-6:
This issue causes openQA test run failure https://openqa.suse.de/tests/14286723 @okurz
I think this failure is caused by incorrect setting of test suite which was triggered mistakenly somehow. But anyway I think this issue should be fixed.
Updated by ybonatakis about 2 months ago · Edited
If this is used to work before, the easiest mitigation could be to switch to apache. However this endpoint doesnt seem to work on my local instance even with apache
Updated by okurz about 2 months ago
- Related to action #130636: high response times on osd - Try nginx on OSD size:S added
Updated by mkittler about 2 months ago
- Status changed from New to In Progress
- Assignee set to mkittler
Updated by mkittler about 2 months ago
Updated by okurz about 2 months ago · Edited
- Subject changed from [openQA][assets] Access to openQA assets forbidden to [openQA][assets] Access to openQA assets forbidden auto_review:"Download (curl) error for 'http://openqa.suse.de/":retry
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1181 merged and deployed.
And called
export host=openqa.suse.de; failed_since="'2024-05-08'" ./openqa-monitor-investigation-candidates | ./openqa-label-known-issues-multi
Updated by livdywan about 2 months ago
mkittler wrote in #note-13:
MR: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1181
https://github.com/os-autoinst/openQA/pull/5636 to update our example in the docs accordingly
Updated by okurz about 2 months ago
- Subject changed from [openQA][assets] Access to openQA assets forbidden auto_review:"Download (curl) error for 'http://openqa.suse.de/":retry to [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry
Updated by okurz about 2 months ago · Edited
Seems like no further jobs were labeled. Rerunning with generalized regex to get rid of potentially problematic '()' and the "'"
Plus
export host=openqa.suse.de; export failed_since="'2024-05-08'"; for i in $(./openqa-monitor-investigation-candidates | sed 's@^[^0-9]*\([0-9]*\) .*$@\1@g'); do openqa-cli api --osd -X post jobs/$i/duplicate ; done
And for jobs that now are labeled with the current ticket openqa-query-for-job-label poo#160171 yields:
14288091|2024-05-13 12:12:04|done|failed|select_modules_and_patterns+registration||grenache-1
14288093|2024-05-13 11:39:27|done|failed|ha_textmode_skip_registration||grenache-1
14287663|2024-05-13 07:33:20|done|incomplete|online_upgrade_sles15sp5_vmware|abandoned: associated worker sapworker2:7 re-connected but abandoned the job|sapworker2
14279413|2024-05-13 07:15:07|done|failed|sev-es-gi-guest_sles15sp5-on-host_developing-kvm||imagetester
14279434|2024-05-12 10:50:16|done|failed|prepare-host_developing-xen||worker35
14281144|2024-05-11 18:36:10|done|failed|SAPHanaSR_ScaleUp_PerfOpt_WMP_node01||worker35
14279360|2024-05-11 17:37:53|done|failed|create_hdd_textmode||worker33
14281003|2024-05-11 17:21:27|done|failed|fips_env_mode_tests_crypt_tool||worker31
14281009|2024-05-11 17:20:21|done|failed|fips_ker_mode_tests_crypt_tool||worker33
14281246|2024-05-11 16:56:09|done|failed|jeos-ltp-commands||worker34
Updated by okurz about 2 months ago
- Due date set to 2024-05-27
- Status changed from In Progress to Feedback
- Priority changed from Urgent to Normal
this should reduce urgency. I assume you don't plan further implementation so we can monitor. I suggest to review the scenarios linked to the labeled failed jobs before resolving so that this ticket wouldn't be reopened afterwards.
Updated by mkittler about 2 months ago · Edited
I checked some scenarios. Two scenarios are still failing:
- https://openqa.suse.de/tests/latest?arch=ppc64le&distri=sle&flavor=Full&machine=ppc64le-spvm&test=select_modules_and_patterns%2Bregistration&version=15-SP6#step/perform_installation/265
- https://openqa.suse.de/tests/latest?arch=ppc64le&distri=sle&flavor=Full&machine=ppc64le-spvm&test=ha_textmode_skip_registration&version=15-SP6#step/perform_installation/265
Maybe #160239 was still causing these tests to fail (considering https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1188 was merged only 24 minutes ago).
Right now I can download e.g. https://openqa.suse.de/assets/repo/SLE-15-SP6-Full-ppc64le-Build88.1-Media1/Module-Basesystem/ppc64le/keyutils-1.6.3-5.6.1.ppc64le.rpm just fine and listing e.g. https://openqa.suse.de/assets/repo/SLE-15-SP6-Full-ppc64le-Build88.1-Media1 works as well. (So not sure why these tests fails due to /var/log/YaST2/y2log-1.gz:2024-05-13 17:28:13 <5> grenache-7(3908) [zypp-core] Exception.cc(log):202 MediaCurl.cc(getFileCopy):795 RETHROW: Downloaded data exceeded the expected filesize '117.6 KiB' of 'http://openqa.suse.de/assets/repo/SLE-15-SP6-Full-ppc64le-Build88.1-Media1/Module-Basesystem/ppc64le/keyutils-1.6.3-5.6.1.ppc64le.rpm'..)
The majority of scenarios look good now:
- https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=svirt-vmware70&test=online_upgrade_sles15sp5_vmware&version=15-SP6
- https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=64bit-ipmi-amd-zen3&test=sev-es-gi-guest_sles15sp5-on-host_developing-kvm&version=15-SP6
- https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=64bit-ipmi-large-mem&test=prepare-host_developing-xen&version=15-SP6
- https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Online&machine=64bit-sap&test=SAPHanaSR_ScaleUp_PerfOpt_WMP_node01&version=15-SP6
- https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Online&machine=s390x-kvm&test=create_hdd_textmode&version=15-SP6#
- https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=JeOS-for-kvm-and-xen&machine=uefi-virtio-vga&test=jeos-ltp-commands&version=15-SP6
Some scenarios don't seem to be relevant:
Updated by okurz about 2 months ago
mkittler wrote in #note-21:
I checked some scenarios. Two scenarios are still failing:
[…]
Maybe #160239 was still causing these tests to fail (considering https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1188 was merged only 24 minutes ago).
yes, my mistake.
Updated by waynechen55 about 2 months ago · Edited
I can access to assets folder on openqa.suse.de now.
Updated by livdywan about 2 months ago
- Subject changed from [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry to [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry size:s
Updated by okurz about 2 months ago
- Subject changed from [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry size:s to [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry size:S
Updated by okurz about 1 month ago
$ openqa-query-for-job-label poo#160171
14337364|2024-05-17 07:49:45|done|parallel_failed|cc_net_case_client||worker29
14337335|2024-05-17 07:38:38|done|failed|cc_net_case_server||worker-arm1
14336776|2024-05-17 06:21:38|done|parallel_failed|cc_net_case_client_uefi||worker29
14336775|2024-05-17 06:21:37|done|failed|cc_net_case_server_uefi||worker34
14336773|2024-05-17 06:19:41|done|failed|cc_net_case_server||worker31
14336778|2024-05-17 06:18:35|done|failed|cc_net_case_client||worker-arm1
14336781|2024-05-17 06:17:32|done|failed|cc_audit_remote_server||worker-arm2
14336767|2024-05-17 06:14:44|done|failed|cc_audit_remote_client_uefi||worker40
14336766|2024-05-17 06:14:44|done|failed|cc_audit_remote_server_uefi||worker31
14336764|2024-05-17 06:14:35|done|failed|cc_audit_remote_server||worker30
Updated by mkittler about 1 month ago
- Subject changed from [openQA][assets] Access to openQA assets forbidden auto_review:"Download.*curl.*error for.*http://openqa.suse.de/":retry size:S to [openQA][assets] Access to openQA assets forbidden size:S
- Status changed from Feedback to Resolved
None of those jobs are relevant. They failed due to:
Error message: Could not resolve host: openqa.suse.de
That error has a completely different cause. Considering I fixed the serving of assets we can be sure that this issue is no longer the cause of download errors so I'm removing the auto review regex.
At this point too many jobs have been mislabeled to remove labels of relevant jobs manually. So I just ran delete from comments where text like '%poo#160171%'; to delete all comments referencing this issue.
With that I would also consider this ticket resolved. (The scenarios mentioned in #160171#note-21 look still good.)