Looks like there are many successful download jobs for those GNOME related assets (and jobs successfully using them, e.g. https://openqa.opensuse.org/tests/2829896). So at least not all jobs are affected.
I've actually tested the error handling locally as part of https://github.com/os-autoinst/openQA/pull/4844. I've just tested it again. The job is definitely not assigned to a worker before the background task has been completed. And once the download fails one gets:
Result: incomplete, finished less than a minute ago (0)
Reason: preparation failed: Downloading "http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/GNOME_Next.x86_64-43.0-Build22.140.iso" failed with: Download of "/hdd/openqa-devel/openqa/share/factory/iso/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20200803-Media.iso" failed: 404 Not Found
So the error case we have in production must be something different.
Looks like in these production jobs we have the following case:
---
args:
- http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/GNOME_Next.x86_64-43.0-Build22.139.iso
- - /var/lib/openqa/share/factory/iso/GNOME_Next.x86_64-43.0-Build22.139.iso
- 0
attempts: 1
children: []
created: 2022-10-21T15:53:01.682181Z
delayed: 2022-10-21T15:53:01.682181Z
expires: ~
finished: 2022-10-21T15:53:44.863466Z
id: 1859732
lax: 0
notes:
gru_id: 18732285
parents: []
priority: 10
queue: default
result: 'Downloading "http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/GNOME_Next.x86_64-43.0-Build22.139.iso"
failed with: Size of "/var/lib/openqa/share/factory/iso/GNOME_Next.x86_64-43.0-Build22.139.iso"
differs, expected 1.4 GiB but downloaded 85 MiB'
retried: ~
retries: 0
started: 2022-10-21T15:53:01.689511Z
state: finished
task: download_asset
time: 2022-10-24T10:50:11.490046Z
worker: 1223
(from https://openqa.opensuse.org/minion/jobs?id=1859732 with e.g. https://openqa.opensuse.org/tests/2826029 as corresponding openQA job)
So there's a download error that is not correctly propagated as such and therefore the job is not ending up as incomplete before being scheduled and we instead get the download error from the cache service. I'm lowering the prio because those jobs will be incompletes either way. Of course I'll fix the error handling on our side.
Note that when trying to download one of those ISOs with Firefox I ran into similar download issues (download ends with less bytes transferred than expected, repeatedly without my internet connection being at fault). So our download code itself doesn't seem to be at fault here at least.