Project

General

Profile

Actions
Copy link

action #44885

closed

Cache service hiccups - Assets are deleted after they are downloaded

Added by szarate over 5 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Category:
Regressions/Crashes
Target version:
-
Start date:
2018-12-07
Due date:
% Done:

0%

Estimated time:

Description

Zaoliang started to report this today, I started to reproduce it...

http://phobos.suse.de/tests/1746808/file/autoinst-log.txt

Basically the assets are downloaded as soon as the next download job is executed... 

[2018-12-07T16:04:48.0077 CET] [info] [pid:32703] +++ setup notes +++
[2018-12-07T16:04:48.0077 CET] [info] [pid:32703] start time: 2018-12-07 15:04:48
[2018-12-07T16:04:48.0077 CET] [info] [pid:32703] running on phobos:1 (Linux 4.18.15-1-default #1 SMP PREEMPT Thu Oct 18 08:56:17 UTC 2018 (5a53676) x86_64)
[2018-12-07T16:04:48.0095 CET] [debug] [pid:32703] Downloading SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2 - request sent to Cache Service.
[2018-12-07T16:04:53.0132 CET] [debug] [pid:32703] Download of SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2 processed
[2018-12-07T16:04:53.0135 CET] [debug] [pid:32703] [INFO] CACHE: Purging non registered /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: removed /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] OpenQA::Worker::Cache: Initialized with localhost at /srv/data/openqa/development_basedir/openqa/cache, current size is 39494643200
[INFO] Downloading SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2 from http://phobos.suse.de/tests/1746807/asset/hdd/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] CACHE: updating the /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2 with "5a5d0000-57c6a30673100" and 1516044288
[DEBUG] CACHE: Asset download successful to /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2, Cache size is: 41010687488

[2018-12-07T16:04:53.0307 CET] [debug] [pid:32703] Downloading SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso - request sent to Cache Service.
[2018-12-07T16:04:58.0337 CET] [debug] [pid:32703] Download of SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso processed
[2018-12-07T16:04:58.0340 CET] [debug] [pid:32703] [INFO] CACHE: Purging non registered /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2
[DEBUG] CACHE: removed /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome.qcow2
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] OpenQA::Worker::Cache: Initialized with localhost at /srv/data/openqa/development_basedir/openqa/cache, current size is 39494643200
[INFO] Downloading SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso from http://phobos.suse.de/tests/1746807/asset/iso/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] CACHE: updating the /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso with "29000000-57c696ac10c00" and 687865856
[DEBUG] CACHE: Asset download successful to /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso, Cache size is: 40182509056

[2018-12-07T16:04:58.0352 CET] [debug] [pid:32703] Downloading SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2 - request sent to Cache Service.
[2018-12-07T16:05:03.0398 CET] [debug] [pid:32703] Download of SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2 processed
[2018-12-07T16:05:03.0407 CET] [debug] [pid:32703] [INFO] CACHE: Purging non registered /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: removed /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLE-15-SP1-Installer-DVD-x86_64-Build110.1-Media1.iso
[DEBUG] CACHE: Health: Real size: 39494643200, Configured limit: 53687091200
[INFO] OpenQA::Worker::Cache: Initialized with localhost at /srv/data/openqa/development_basedir/openqa/cache, current size is 39494643200
[INFO] Downloading SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2 from http://phobos.suse.de/tests/1746807/asset/hdd/SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2
[DEBUG] CACHE: Download of /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2 failed with: 404 - Not Found
[DEBUG] CACHE: requested to remove nonexisting asset /srv/data/openqa/development_basedir/openqa/cache/phobos.suse.de/SLES-15-SP1-x86_64-Build110.1@64bit-gnome-uefi-vars.qcow2

[2018-12-07T16:05:08.0459 CET] [info] [pid:32703] rsync: receiving incremental file list

sent 769 bytes  received 824,876 bytes  1,651,290.00 bytes/sec
total size is 3,795,211,052  speedup is 4,596.66

Related issues 1 (0 open — 1 closed)

Blocked by openQA Project - action #28328: job was triggered trying to download HDD image but it's already goneRejectedokurz2017-11-24

Actions
Actions
Copy link
 #1

Updated by szarate over 5 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by szarate over 5 years ago

This seems to happen when the job has two assets (HDD or ISO)

Actions
Copy link
 #3

Updated by szarate over 5 years ago

  • Subject changed from Cache service hickups - Assets are deleted after they are downloaded to Cache service hiccups - Assets are deleted after they are downloaded
Actions
Copy link
 #4

Updated by szarate over 5 years ago

  • Status changed from New to Feedback
  • Priority changed from Urgent to Normal

For some reason, deleting the CACHEDIRECTORY. helps :). http://phobos.suse.de/tests/1746836

Setting to fedback for the time being. It might be a bit annoying but life is not easy, I just wonder why deleting the old cache fixes this...

Actions
Copy link
 #5

Updated by okurz over 4 years ago

  • Blocked by action #28328: job was triggered trying to download HDD image but it's already gone added
Actions
Copy link
 #6

Updated by okurz over 4 years ago

  • Status changed from Feedback to Blocked
  • Assignee set to okurz

that suspiciously looks like #28328

Actions
Copy link
 #7

Updated by okurz about 4 years ago

  • Status changed from Blocked to Resolved

With all the recent cache service rework and especially also with #46742 I am confident the problem is solved – except for a potential minor case when the job group size limit is not even able to accomodate the required assets which is not the fault of the cache service but needs to be handled by the users accordingly.

Actions
Copy link

Also available in: Atom PDF