action #28328

job was triggered trying to download HDD image but it's already gone

Added by okurz over 2 years ago. Updated about 1 month ago.

Status:RejectedStart date:24/11/2017
Priority:NormalDue date:
Assignee:okurz% Done:

0%

Category:Concrete Bugs
Target version:-
Difficulty:
Duration:

Description

Observation

https://openqa.suse.de/tests/1269748/file/autoinst-log.txt

start time: 2017-11-24 07:09:39
…
CACHE: Download of /var/lib/openqa/cache/SLES-15-aarch64-349.1@aarch64-minimal_with_sdk349.1_installed.qcow2 failed with: 404 - Not Found
+++ worker notes +++
end time: 2017-11-24 07:09:40
result: setup failure: Can't download SLES-15-aarch64-349.1@aarch64-minimal_with_sdk349.1_installed.qcow2

and from the parent:

end time: 2017-11-23 17:20:51
uploading install_and_reboot-y2logs.tar.bz2
uploading SLES-15-aarch64-349.1@aarch64-minimal_with_sdk349.1_installed.qcow2
Checksum comparison (actual:expected) 1032847561:1032847561 with size (actual:expected) 836435968:836435968

osd:openqa-gru:

[Fri Nov 24 02:53:09 2017] [28989:info] GRU: removing /var/lib/openqa/share/factory/hdd/SLES-15-aarch64-349.1@aarch64-minimal_with_sdk349.1_installed.qcow2

So it was deleted as expected after the parent job uploaded it but before the downstream job had a chance to act on it.

Problem

Isn't the asset being marked as "used" by a scheduled job to prevent GRU from cleaning that up?


Related issues

Related to openQA Project - action #19672: GRU may delete assets while jobs are registered Resolved 08/06/2017
Related to openQA Project - action #16496: [tools][sprint 201711.2] display current disk space consu... Resolved 06/02/2017
Related to openQA Tests - action #25380: [sle][functional][epic] test fails in install - tries to ... Resolved 05/12/2017 27/02/2018
Related to openQA Project - action #34783: Don't let jobs incomplete if mandatory resources are missing In Progress 12/04/2018
Related to openQA Project - action #12180: [webui] Prevent tests to be triggered when required asset... New 31/05/2016
Blocks openQA Project - action #44885: Cache service hiccups - Assets are deleted after they are... Resolved 07/12/2018

History

#1 Updated by coolo over 2 years ago

If it's used or not doesn't matter - if GRU deletes it, the job group was obviously not big enough to hold the working set. You have 100G for that group and the isos alone are around 70

But there is some subtle bug hidden, because SLES-15-x86_64-305.1-minimal_with_sdk305.1_installed.qcow2 is still present, but is 5 weeks old.

https://progress.opensuse.org/issues/19672#note-8 might be part of the puzzle
as is https://progress.opensuse.org/issues/16496 - it's just too hard atm to admin the job group sizes.

#2 Updated by coolo over 2 years ago

I increased the job group size of functional group to 500GB, 100GB was a bit lightweight

#3 Updated by okurz over 2 years ago

  • Related to action #19672: GRU may delete assets while jobs are registered added

#4 Updated by okurz over 2 years ago

  • Related to action #16496: [tools][sprint 201711.2] display current disk space consumption of job groups added

#5 Updated by okurz over 2 years ago

I guess I would be fine if the job would have been canceled with a message/state/comment instead of incomplete.

#6 Updated by SLindoMansilla over 2 years ago

  • Related to action #25380: [sle][functional][epic] test fails in install - tries to install SLE12 packages -> update test for sle15 added

#7 Updated by AdamWill over 2 years ago

I have thought for a while that the cleanup code should avoid deleting assets associated with pending/running jobs...

#8 Updated by okurz about 1 year ago

  • Related to action #34783: Don't let jobs incomplete if mandatory resources are missing added

#9 Updated by okurz 5 months ago

  • Related to action #12180: [webui] Prevent tests to be triggered when required assets are not present (anymore) added

#10 Updated by okurz 5 months ago

  • Blocks action #44885: Cache service hiccups - Assets are deleted after they are downloaded added

#11 Updated by okurz about 1 month ago

  • Status changed from New to Rejected
  • Assignee set to okurz

I guess by now we have changed the asset cleanup and quota management code enough again to call the behaviour currently described in this ticket as design. The alternative of locking assets for currently scheduled jobs even though job group quota is exceeded sounds dangerous as well. We could try to delete assets linked to not-unfinished jobs first but that there are also good arguments to prefer to keep assets of finished jobs so I don't think we should even make that call.

Also available in: Atom PDF