QA » openQA Project

Got an update from infra: If possible perhaps by tomorrow we might get an answer.

I got this also but is unrelated to /var/lib/openqa as the logs are written to /var/log/openqa_scheduler

Aug 23 12:46:23 openqa openqa-scheduler[23341]: Uncaught exception from user code:
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Can't write to log: No space left on device at /usr/share/openqa/script/../lib/OpenQA/Utils.pm line 326.
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Mojo::Log::append('Mojo::Log=HASH(0x210f370)', '[2018-08-23T12:46:23.0687 CEST] [debug] [pid:23341] Could not...') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Log.pm line 70
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Mojo::Log::_message('Mojo::Log=HASH(0x210f370)', 'debug', '[pid:23341] Could not get new jobs to allocate: {UNKNOWN}: Ca...') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/EventE
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Mojo::EventEmitter::emit('Mojo::Log=HASH(0x210f370)', 'message', 'debug', '[pid:23341] Could not get new jobs to allocate: {UNKNOWN}: Ca...') called at /usr/lib/perl5/vendor_perl/5.
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Mojo::Log::_log('Mojo::Log=HASH(0x210f370)', 'debug', '[pid:23341] Could not get new jobs to allocate: {UNKNOWN}: Ca...') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Log.pm lin
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Mojo::Log::debug('Mojo::Log=HASH(0x210f370)', '[pid:23341] Could not get new jobs to allocate: {UNKNOWN}: Ca...') called at /usr/share/openqa/script/../lib/OpenQA/Utils.pm line 326
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         OpenQA::Utils::_log('debug', '[pid:23341] Could not get new jobs to allocate: {UNKNOWN}: Ca...') called at /usr/share/openqa/script/../lib/OpenQA/Utils.pm line 312
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         OpenQA::Utils::_log_msg('debug', 'Could not get new jobs to allocate: {UNKNOWN}: Can\'t write t...', 'channels', 'ARRAY(0x327aba8)', 'standard', 1) called at /usr/share/openqa/scrip
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         OpenQA::Utils::_log_msg('debug', 'Could not get new jobs to allocate: {UNKNOWN}: Can\'t write t...') called at /usr/share/openqa/script/../lib/OpenQA/Utils.pm line 251
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         OpenQA::Utils::log_debug('Could not get new jobs to allocate: {UNKNOWN}: Can\'t write t...') called at /usr/share/openqa/script/../lib/OpenQA/Scheduler/Scheduler.pm line 290
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         OpenQA::Scheduler::Scheduler::catch {...} ('DBIx::Class::Exception=HASH(0x99e9d10)') called at /usr/lib/perl5/vendor_perl/5.18.2/Try/Tiny.pm line 115
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Try::Tiny::try('CODE(0x9538b28)', 'Try::Tiny::Catch=REF(0x9278618)') called at /usr/share/openqa/script/../lib/OpenQA/Scheduler/Scheduler.pm line 293
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         OpenQA::Scheduler::Scheduler::schedule() called at /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/Net/DBus/Callback.pm line 119
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Net::DBus::Callback::invoke('Net::DBus::Callback=HASH(0xc48a300)') called at /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/Net/DBus/Reactor.pm line 385
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Net::DBus::Reactor::step('Net::DBus::Reactor=HASH(0x210ee78)') called at /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/Net/DBus/Reactor.pm line 325
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         Net::DBus::Reactor::run('Net::DBus::Reactor=HASH(0x210ee78)') called at /usr/share/openqa/script/../lib/OpenQA/Scheduler.pm line 102
Aug 23 12:46:23 openqa openqa-scheduler[23341]:         OpenQA::Scheduler::run() called at /usr/share/openqa/script/openqa-scheduler line 38

I don't know a better way than calling du on /var/lib/openqa to find out where the space is used, if it's assets, test results or something else that was unforeseen.

I'm guessing it's matter of properly calculating, because doing a simple du can run for 12 hours and get nothing :)

I'm not sure how important this one is, but I have removed hdd/fixed/SLE-12-SP3-Server-Build0450-allpatterns-for-balance-old.qcow2 from there, got it on my local machine (will make it available later)

openqa:/var/lib/openqa # du -chs share/factory/*
1.5T    share/factory/hdd
399G    share/factory/iso
1.8G    share/factory/other
766G    share/factory/repo
7.7M    share/factory/tmp
2.6T    total

Current space is 9.0T, of which we have 6.9T being used (that's about 76%), that leaves us with 4.3T in just test results... We need to think about or retention policy, I already made Sebastian aware of it.