Project

General

Profile

action #44885

Cache service hiccups - Assets are deleted after they are downloaded

Added by szarate over 1 year ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
-
Start date:
2018-12-07
Due date:
% Done:

0%

Estimated time:
Difficulty:
Duration:

Description

Zaoliang started to report this today, I started to reproduce it...

http://phobos.suse.de/tests/1746808/file/autoinst-log.txt

Basically the assets are downloaded as soon as the next download job is executed...

[2018-12-07T16:04:48.0077 CET] [info] [pid:32703] +++ setup notes +++
[2018-12-07T16:04:48.0077 CET] [info] [pid:32703] start time: 2018-12-07 15:04:48
[2018-12-07T16:04:48.0077 CET] [info] [pid:32703] running on phobos:1 (Linux 4.18.15-1-default #1 SMP PREEMPT Thu Oct 18 08:56:17 UTC 2018 (5a53676) x86_64)
[2018-12-07T16:04:48.0095 CET] [debug] [pid:32703] Downloading SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2 - request sent to Cache Service.
[2018-12-07T16:04:53.0132 CET] [debug] [pid:32703] Download of SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2 processed
[2018-12-07T16:04:53.0135 CET] [debug] [pid:32703] [INFO] CACHE: Purging non registered /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: removed /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] OpenQA::Worker::Cache: Initialized with localhost at /srv/data/openqa/development_basedir/openqa/cache, current size is 39494643200
[INFO] Downloading SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2 from http://phobos.suse.de/tests/1746807/asset/hdd/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] CACHE: updating the /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2 with "5a5d0000-57c6a30673100" and 1516044288
[DEBUG] CACHE: Asset download successful to /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2, Cache size is: 41010687488

[2018-12-07T16:04:53.0307 CET] [debug] [pid:32703] Downloading SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso - request sent to Cache Service.
[2018-12-07T16:04:58.0337 CET] [debug] [pid:32703] Download of SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso processed
[2018-12-07T16:04:58.0340 CET] [debug] [pid:32703] [INFO] CACHE: Purging non registered /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2
[DEBUG] CACHE: removed /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] OpenQA::Worker::Cache: Initialized with localhost at /srv/data/openqa/development_basedir/openqa/cache, current size is 39494643200
[INFO] Downloading SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso from http://phobos.suse.de/tests/1746807/asset/iso/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] CACHE: updating the /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso with "29000000-57c696ac10c00" and 687865856
[DEBUG] CACHE: Asset download successful to /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso, Cache size is: 40182509056

[2018-12-07T16:04:58.0352 CET] [debug] [pid:32703] Downloading SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2 - request sent to Cache Service.
[2018-12-07T16:05:03.0398 CET] [debug] [pid:32703] Download of SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2 processed
[2018-12-07T16:05:03.0407 CET] [debug] [pid:32703] [INFO] CACHE: Purging non registered /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: removed /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] OpenQA::Worker::Cache: Initialized with localhost at /srv/data/openqa/development_basedir/openqa/cache, current size is 39494643200
[INFO] Downloading SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2 from http://phobos.suse.de/tests/1746807/asset/hdd/SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2
[DEBUG] CACHE: Download of /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2 failed with: 404 - Not Found
[DEBUG] CACHE: requested to remove nonexisting asset /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2

[2018-12-07T16:05:08.0459 CET] [info] [pid:32703] rsync: receiving incremental file list

sent 769 bytes  received 824,876 bytes  1,651,290.00 bytes/sec
total size is 3,795,211,052  speedup is 4,596.66

Related issues

Blocked by openQA Project - action #28328: job was triggered trying to download HDD image but it's already goneRejected2017-11-24

History

#1 Updated by szarate over 1 year ago

  • Description updated (diff)

#2 Updated by szarate over 1 year ago

This seems to happen when the job has two assets (HDD or ISO)

#3 Updated by szarate over 1 year ago

  • Subject changed from Cache service hickups - Assets are deleted after they are downloaded to Cache service hiccups - Assets are deleted after they are downloaded

#4 Updated by szarate over 1 year ago

  • Status changed from New to Feedback
  • Priority changed from Urgent to Normal

For some reason, deleting the CACHEDIRECTORY. helps :). http://phobos.suse.de/tests/1746836

Setting to fedback for the time being. It might be a bit annoying but life is not easy, I just wonder why deleting the old cache fixes this...

#5 Updated by okurz 7 months ago

  • Blocked by action #28328: job was triggered trying to download HDD image but it's already gone added

#6 Updated by okurz 7 months ago

  • Status changed from Feedback to Blocked
  • Assignee set to okurz

that suspiciously looks like #28328

#7 Updated by okurz 3 months ago

  • Status changed from Blocked to Resolved

With all the recent cache service rework and especially also with #46742 I am confident the problem is solved – except for a potential minor case when the job group size limit is not even able to accomodate the required assets which is not the fault of the cache service but needs to be handled by the users accordingly.

Also available in: Atom PDF