action #40862

Out of disk space killed the webui (on osd)

Added by coolo over 1 year ago. Updated over 1 year ago.

Status:RejectedStart date:11/09/2018
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Difficulty:
Duration:

Description

What's worrysome about this failure: none of the partitions is over 80% - and if this was just a temporary
hickup, the webui getting killed because of log files sounds evil.

``
Sep 11 12:39:46 openqa openqa[15105]: Failure of user callbacks dispatched using the G_KEEPERR flag could
Sep 11 12:39:46 openqa openqa[15105]: also result in this warning. See "G_KEEPERR" in perlcall.
Sep 11 12:39:46 openqa openqa[15105]:

Sep 11 12:39:46 openqa openqa[15105]: (in cleanup) Can't write to log: No space left on device at
Sep 11 12:39:46 openqa openqa[15105]: /usr/share/openqa/script/../lib/OpenQA/Utils.pm line 326 (#1)
Sep 11 12:39:46 openqa openqa[15105]: (W misc) This prefix usually indicates that a DESTROY() method raised
Sep 11 12:39:46 openqa openqa[15105]: the indicated exception. Since destructors are usually called by the
Sep 11 12:39:46 openqa openqa[15105]: system at arbitrary points during execution, and often a vast number of
Sep 11 12:39:46 openqa openqa[15105]: times, the warning is issued only once for any number of failures that
Sep 11 12:39:46 openqa openqa[15105]: would otherwise result in the same message being repeated.
Sep 11 12:39:46 openqa openqa[15105]:

Sep 11 12:39:46 openqa openqa[15105]: Failure of user callbacks dispatched using the G_KEEPERR flag could
Sep 11 12:39:46 openqa openqa[15105]: also result in this warning. See "G_KEEPERR" in perlcall.
Sep 11 12:39:46 openqa openqa[15105]:

Sep 11 12:39:46 openqa openqa[15105]: Uncaught exception from user code:
Sep 11 12:39:46 openqa openqa[15105]: Can't write to log: No space left on device at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Server/Prefork.pm line 162.
Sep 11 12:39:46 openqa openqa[15105]: Mojo::Log::append('Mojo::Log=HASH(0x80cb2a8)', '[2018-09-11T12:39:46.0972 CEST] [info] Worker 24893 started\x{a}') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Log.pm l
Sep 11 12:39:46 openqa openqa[15105]: Mojo::Log::_message('Mojo::Log=HASH(0x80cb2a8)', 'info', 'Worker 24893 started') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/EventEmitter.pm line 15
Sep 11 12:39:46 openqa openqa[15105]: Mojo::EventEmitter::emit('Mojo::Log=HASH(0x80cb2a8)', 'message', 'info', 'Worker 24893 started') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Log.pm line 58
Sep 11 12:39:46 openqa openqa[15105]: Mojo::Log::_log('Mojo::Log=HASH(0x80cb2a8)', 'info', 'Worker 24893 started') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Log.pm line 42
Sep 11 12:39:46 openqa openqa[15105]: Mojo::Log::info('Mojo::Log=HASH(0x80cb2a8)', 'Worker 24893 started') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Server/Prefork.pm line 162
Sep 11 12:39:46 openqa openqa[15105]: Mojo::Server::Prefork::_spawn('Mojo::Server::Prefork=HASH(0x9b544c0)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Server/Prefork.pm line 100
Sep 11 12:39:46 openqa openqa[15105]: Mojo::Server::Prefork::_manage('Mojo::Server::Prefork=HASH(0x9b544c0)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Server/Prefork.pm line 85
Sep 11 12:39:46 openqa openqa[15105]: Mojo::Server::Prefork::run('Mojo::Server::Prefork=HASH(0x9b544c0)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Command/prefork.pm line 31
Sep 11 12:39:46 openqa openqa[15105]: Mojolicious::Command::prefork::run('Mojolicious::Command::prefork=HASH(0x9b50c60)', '--proxy', '-i', 100, '-H', 400, '-w', 40, '-G', ...) called at /usr/lib/perl5/vendor_pe
Sep 11 12:39:46 openqa openqa[15105]: Mojolicious::Commands::run('Mojolicious::Commands=HASH(0x86e16f8)', 'prefork', '-m', 'production', '--proxy', '-i', 100, '-H', 400, ...) called at /usr/lib/perl5/vendor_per
Sep 11 12:39:46 openqa openqa[15105]: Mojolicious::start('OpenQA::WebAPI=HASH(0x1463818)') called at /usr/lib/perl5/vendor_perl/5.18.2/Mojolicious/Commands.pm line 71
Sep 11 12:39:46 openqa openqa[15105]: Mojolicious::Commands::start_app('Mojolicious::Commands', 'OpenQA::WebAPI') called at /usr/share/openqa/script/../lib/OpenQA/WebAPI.pm line 497
Sep 11 12:39:46 openqa openqa[15105]: OpenQA::WebAPI::run() called at /usr/share/openqa/script/openqa line 34
Sep 11 12:39:46 openqa openqa[15105]: (in cleanup) Can't write to log: No space left on device at
Sep 11 12:39:46 openqa openqa[15105]: /usr/share/openqa/script/../lib/OpenQA/Utils.pm line 326 (#1)
Sep 11 12:39:46 openqa openqa[15105]: (W misc) This prefix usually indicates that a DESTROY() method raised
Sep 11 12:39:46 openqa openqa[15105]: the indicated exception. Since destructors are usually called by the
Sep 11 12:39:46 openqa openqa[15105]: system at arbitrary points during execution, and often a vast number of
Sep 11 12:39:46 openqa openqa[15105]: times, the warning is issued only once for any number of failures that
Sep 11 12:39:46 openqa openqa[15105]: would otherwise result in the same message being repeated.
Sep 11 12:39:46 openqa openqa[15105]:

Sep 11 12:39:46 openqa openqa[15105]: Failure of user callbacks dispatched using the G_KEEPERR flag could
Sep 11 12:39:46 openqa openqa[15105]: also result in this warning. See "G_KEEPERR" in perlcall.
Sep 11 12:39:46 openqa openqa[15105]:

Sep 11 12:39:47 openqa openqa[15105]: (in cleanup) Can't write to log: No space left on device at
Sep 11 12:39:47 openqa openqa[15105]: /usr/share/openqa/script/../lib/OpenQA/Utils.pm line 326 (#1)
Sep 11 12:39:47 openqa openqa[15105]: (W misc) This prefix usually indicates that a DESTROY() method raised
Sep 11 12:39:47 openqa systemd[1]: openqa-webui.service: Main process exited, code=exited, status=4/NOPERMISSION


Related issues

Related to openQA Project - action #34042: [tools] Worker goes to infinite loop during upload of scr... Resolved 05/04/2018
Duplicates openQA Project - action #34282: implement catching errors in all our log_* functions Resolved 05/04/2018

History

#1 Updated by EDiGiacinto over 1 year ago

(sadly) the same thing happens on worker side as well

#2 Updated by EDiGiacinto over 1 year ago

  • Related to action #34042: [tools] Worker goes to infinite loop during upload of screenshots in case of writing failure added

#3 Updated by EDiGiacinto over 1 year ago

  • Related to action #34282: implement catching errors in all our log_* functions added

#4 Updated by coolo over 1 year ago

  • Related to deleted (action #34282: implement catching errors in all our log_* functions)

#5 Updated by coolo over 1 year ago

  • Duplicates action #34282: implement catching errors in all our log_* functions added

#6 Updated by coolo over 1 year ago

  • Status changed from New to Rejected

let's treat it as dup

Also available in: Atom PDF