Project

General

Profile

Actions

action #40223

closed

Document GRU cleanup strategy

Added by riafarov almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2018-08-24
Due date:
% Done:

0%

Estimated time:

Description

Description

We have following job: https://openqa.opensuse.org/tests/741176# started at 8am

However, GRU has removed one of the repos [2018-08-24T01:21:01.0763 UTC] [info] GRU: removing /var/lib/openqa/share/factory/repo/openSUSE-Tumbleweed-oss-aarch64-Snapshot20180820-debuginfo
So job fails. GRU hasn't removed repo in REPO_0 variable though. Assets are not older than 14 days and there is plenty of space available.


Related issues 2 (0 open2 closed)

Related to openQA Tests - action #40256: [opensuse][functional][y] Verify if repo is available before adding it in zypper_ar test moduleResolvedmloviska2018-08-272018-09-25

Actions
Related to openQA Tests - action #40349: [sle][functional][u][migration][sle15sp1] Jobs incomplete due to missing assetsResolvedszarate2018-08-28

Actions
Actions #1

Updated by szarate almost 6 years ago

  • Target version set to Ready

We seem to be having this problem in osd too

Actions #2

Updated by mkittler almost 6 years ago

  • Target version deleted (Ready)

@szarate And my PR which might affect this hasn't even been merged yet. I note this because it might make sense to merge these changes first before trying to fix this issue.

Assets are not older than 14 days and there is plenty of space available.

That limit is only applied to jobs which aren't in a group. But yes, due to a bug it could have been applied here accidentally.

and there is plenty of space available

The job you mention in the ticket summary is in group openSUSE Tumbleweed AArch64 where 78 GiB of 80 GiB are used. I wouldn't say that 2 GiB is 'plenty of space'. Or do you think the value of 78 GiB is wrongly computed?

Actions #3

Updated by riafarov almost 6 years ago

@mkittler I was not aware of that setting to be honest, I meant disk space available. So we can actually increase space for it. However, still 80GB is enough to keep latest assets at least and not remove some of them.

Actions #4

Updated by mkittler almost 6 years ago

  • Target version set to Ready

However, still 80GB is enough to keep latest assets at least and not remove some of them.

Yes, it should of course remove preferably old assets and removing assets from pending jobs should be prevented at all. So yes, it is a bug. (I removed the target version only by mistake.)

It would be interesting whether this still occurs after my optimization/refactoring. When doing that, I also added some more tests (which, however are only done with a few assets from our fixtures).

Actions #5

Updated by riafarov almost 6 years ago

  • Related to action #40256: [opensuse][functional][y] Verify if repo is available before adding it in zypper_ar test module added
Actions #6

Updated by szarate almost 6 years ago

  • Related to action #40349: [sle][functional][u][migration][sle15sp1] Jobs incomplete due to missing assets added
Actions #7

Updated by coolo almost 6 years ago

  • Subject changed from [tools] gru cleans up assets too early to Document GRU cleanup strategy
  • Category set to 140
  • Target version changed from Ready to Current Sprint

GRU doesn't care for the age of the asset - only about the age of the job using it. So if there are only old jobs attached to that asset, it will remove it.

So if your job group isn't large enough to hold the current assets - you need to increase it, no way around it. That's why we put so many details in /admin/assets page.

Actions #8

Updated by mkittler almost 6 years ago

  • Assignee set to mkittler
Actions #9

Updated by mkittler almost 6 years ago

  • Status changed from New to In Progress
Actions #10

Updated by mkittler almost 6 years ago

  • Status changed from In Progress to Resolved

PR is merged

Actions #11

Updated by coolo over 5 years ago

  • Target version changed from Current Sprint to Done
Actions

Also available in: Atom PDF