Project

General

Profile

Actions
Copy link

action #97976

open

[alert] OSD file systems - assets

Added by okurz over 3 years ago. Updated about 3 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
QA (public) - future
Start date:
2021-09-02
Due date:
2021-10-01 (over 3 years late)
% Done:

50%

Estimated time:
(Total: 0.00 h)

Description

Observation

https://stats.openqa-monitor.qa.suse.de/d/WebuiDb/webui-summary?tab=alert&viewPanel=74&orgId=1&from=now-24h&to=now shows a constant increase since last midnight and there was no asset cleanup. https://openqa.suse.de/minion/locks shows currently "limit_tasks", "limit_screenshots_task", "limit_results_and_logs_task", "process_job_results_task" all to expire in 10h, and "limit_results_and_logs" currently running since 14 hours. Maybe that blocks (again?) asset cleanup?

Subtasks 4 (2 open — 2 closed)

openQA Project (public) - action #97979: Asset cleanup takes very long to process 60k files in "other" size:MResolvedmkittler2021-09-022021-10-01

Actions
openQA Project (public) - action #99420: Asset cleanup takes very long to process 60k files in "other" - now for real!New

Actions
openQA Project (public) - action #99426: Asset cleanup takes very long to process 60k files in "other" - suboptimal logging?Resolvedokurz

Actions
openQA Project (public) - action #100599: Asset cleanup takes very long to process 60k files in "other" - too verbose logging, switch some debug to trace?New

Actions

Related issues 1 (0 open — 1 closed)

Related to openQA Infrastructure (public) - action #96789: File systems alert 90.256 assets used size:MResolvedmkittler2021-08-12

Actions
Actions
Copy link
 #1

Updated by okurz over 3 years ago

  • Status changed from New to In Progress
  • Assignee set to okurz
Actions
Copy link
 #2

Updated by okurz over 3 years ago

https://openqa.suse.de/minion/jobs?id=2776306 says it was created 14 hours ago and is "inactive", why is that? And what does the runtime "a few seconds delay" mean?

Actions
Copy link
 #3

Updated by okurz over 3 years ago

unlocked "limit_results_and_logs_task", should this block the assets cleanup? Now https://openqa.suse.de/minion/jobs?id=2776306 started. Attaching to the according gru process with strace -f -y … I see a lot of lookup of /var/lib/openqa/share/factory/other. This can take some time as the amount of files there seems to increase.

Monitoring progress on OSD.

Actions
Copy link
 #5

Updated by okurz over 3 years ago

  • Status changed from In Progress to Blocked
Actions
Copy link
 #6

Updated by okurz over 3 years ago

  • Related to action #96789: File systems alert 90.256 assets used size:M added
Actions
Copy link
 #7

Updated by okurz over 3 years ago

#96789 seems to originally describe the same problem.

Actions
Copy link
 #8

Updated by kraih over 3 years ago

okurz wrote:

https://openqa.suse.de/minion/jobs?id=2776306 says it was created 14 hours ago and is "inactive", why is that? And what does the runtime "a few seconds delay" mean?

In case you didn't get an answer somewhere else yet. It means the job was enqueued or retried (retried in this case) with a delay of a few seconds. Probably because it checks the lock, and if it exists retries itself with the delay, over and over until it can get the lock itself.

Actions
Copy link
 #9

Updated by okurz about 3 years ago

  • Status changed from Blocked to New
  • Assignee deleted (okurz)
  • Target version changed from Ready to future
Actions
Copy link

Also available in: Atom PDF