Actions
action #53261
closedtests incomplete with "No space left on device" openqaworker-arm-2 ran out of space in uploading files in jobs -> cache takes 500G, should be only 50G?
Start date:
2019-06-18
Due date:
% Done:
0%
Estimated time:
Description
Observation¶
[18/06/2019 12:17:16] <riafarov> [2019-06-18T10:09:36.107 UTC] [debug] Can't write to file "testresults/logs_from_installation_system-25.txt": No space left on device at /usr/lib/os-autoinst/basetest.pm line 436.
[18/06/2019 12:17:35] <riafarov> seems that openqaworker-arm-2 disk is full
[18/06/2019 12:25:47] <okurz> @riafarov: hm, it's not
[18/06/2019 12:26:41] <okurz> but something seems odd. Don't know where the pool is stored
[18/06/2019 12:28:52] <okurz> @riafarov ok I think I found it, the cache takes up 500G, is there a ticket for that already?
Problem¶
Investigating and comparing the space usage on other workers:
sudo salt '*' cmd.run 'du -sh /var/lib/openqa/cache'
openqaworker-arm-2.suse.de:
4.0K /var/lib/openqa/cache
openqa.suse.de:
du: cannot access '/var/lib/openqa/cache': No such file or directory
powerqaworker-qam-1:
62G /var/lib/openqa/cache
QA-Power8-4-kvm.qa.suse.de:
59G /var/lib/openqa/cache
openqaworker3.suse.de:
58G /var/lib/openqa/cache
openqaw1.qa.suse.de:
59G /var/lib/openqa/cache
openqaworker5.suse.de:
56G /var/lib/openqa/cache
QA-Power8-5-kvm.qa.suse.de:
58G /var/lib/openqa/cache
openqaworker2.suse.de:
59G /var/lib/openqa/cache
openqaw2.qa.suse.de:
57G /var/lib/openqa/cache
openqaworker13.suse.de:
58G /var/lib/openqa/cache
openqaworker6.suse.de:
57G /var/lib/openqa/cache
openqaworker8.suse.de:
59G /var/lib/openqa/cache
openqaworker9.suse.de:
57G /var/lib/openqa/cache
openqaworker7.suse.de:
47G /var/lib/openqa/cache
grenache-1.qa.suse.de:
62G /var/lib/openqa/cache
openqaworker-arm-1.suse.de:
Minion did not return. [Not connected]
malbec.arch.suse.de:
Minion did not return. [Not connected]
ERROR: Minions returned with non-zero exit code
Actions