Caching issue on new snapshots synced to o3 - no cache minion workers available
In the night of Dec 3/4, Snapshot 1203 was synced to openQA and jobs created for execution.
A large number of jobs (roughly 70) seems to have marked 'incomplete' within minutes, as files failed to be cached and the job thus failed to start.
[2018-12-04T03:35:08.0090 CET] [info] result: setup failure: Can't download openSUSE-Tumbleweed-DVD-x86_64-Snapshot20181203-Media.iso to /var/lib/openqa/cache/openqa1-opensuse/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20181203-Media.iso
[2018-12-04T03:35:52.0783 CET] [info] result: setup failure: No workers active in the cache service.
Logs indicate like the two workers might have differentissues; I could not find any incomplete job that would have been dispatched to OW1
#1 Updated by coolo over 4 years ago
https://openqa.opensuse.org/tests/808095 had it downloaded correctly, but was restarted by autodeploy (I assume) and https://openqa.opensuse.org/tests/808154 afterwards has download failures.
So autodeploy left the cache in limbo
#2 Updated by okurz over 4 years ago
- Related to action #44105: if workercache dies, we get *tons* of incompletes added
#3 Updated by szarate over 4 years ago
- Subject changed from Caching issue on new snapshots synced to o3 to Caching issue on new snapshots synced to o3 - no cache minion workers available
#4 Updated by okurz almost 4 years ago
- Category set to Concrete Bugs
#5 Updated by okurz over 3 years ago
- Status changed from New to Resolved
- Assignee set to okurz
could be that #44105 really solved it. I have not seen the problem since then