action #163766
Updated by okurz 5 months ago
## Observation
We are seeing this error and a couple of incompletes today:
```
Date: Thu, 11 Jul 2024 11:51:31 +0000
From: "GitLab@SUSE" <gitlab@suse.de>
To: osd-admins@suse.de
Subject: Scripts CI | Failed pipeline for master | 33a115c3
```
https://gitlab.suse.de/openqa/scripts-ci/-/pipelines/1207977
https://stats.openqa-monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=1720656188837&to=1720699027331&viewPanel=16
https://openqa.suse.de/tests/14894947
```
Result: incomplete, finished 10 minutes ago (ran for 36:46 minutes)
Reason: asset failure: Failed to download SLES-15-SP5-x86_64-mru-install-minimal-with-addons-Build20240710-1-Server-DVD-Updates-64bit.qcow2 to /var/lib/openqa/cache/openqa.suse.de/SLES-15-SP5-x86_64-mru-install-minimal-with-addons-Build20240710-1-Server-DVD-Updates-64bit.qcow2
```
The detailed error message is on e.g. https://openqa.suse.de/tests/14894946:
```
[info] [#21729] Download error 598, waiting 5 seconds for next try (1 remaining)
```
(see https://http.dev/598)
## Suggestions
* Look for accordingly affected jobs, e.g. using openqa-label-known-issues or look into the database and retrigger accordingly
* Due to the timely coincidence this is very related to #163592 but assets should be served by NGINX directly (so the unresponsive Mojo web app should not have an impact)
* Verify that those asset downloads work independently from the web app (e.g. by stopping the web app shortly and try to download the asset)
* Maybe the web app is still involved for some redirection? That would be fine but of course means it is related to #163592.
* Can we split that completely?
* Check how this timeout is handled by NGINX
## Rollback actions
* Set pipeline status emails in https://gitlab.suse.de/openqa/scripts-ci/-/settings/integrations/pipelines_email/edit to "Active"
Back