action #106759
Updated by tinita about 2 years ago
## Observation # /var/log/openqa [2022-02-12T04:12:47.195902Z] [error] Worker 22011 has no heartbeat (400 seconds), restarting [2022-02-12T04:12:56.228360Z] [error] Worker 28596 has no heartbeat (400 seconds), restarting ## Acceptance criteria **AC1**: Heartbeat message is not observed ## Suggestions - ~~Add the message to the blocklist~~ - Look at Mojolicious APIs related to preforking of Mojo workers - ~~Extend the configured timeout from 400s~~ The timeout's already pretty high - Confirm where the errors are logged, and add context - ~~500 error in access_log from live_view_handler~~ (unrelated, time doesn't match and the live view handler doesn't use preforking) `[15/Feb/2022:07:16:02 +0000] "GET /liveviewhandler/tests/2189494/developer/ws-proxy HTTP/1.1" 500 ` ## Additional info from `/usr/lib/perl5/vendor_perl/5.34.0/Mojolicious/Guides/FAQ.pod`: ``` =head2 What does "Worker 31842 has no heartbeat (50 seconds), restarting" mean? As long as they are accepting new connections, worker processes of all built-in pre-forking web servers send heartbeat messages to the manager process at regular intervals, to signal that they are still responsive. A blocking operation such as an infinite loop in your application can prevent this, and will force the affected worker to be restarted after a timeout. This timeout defaults to C<50> seconds and can be extended with the attribute L<Mojo::Server::Prefork/"heartbeat_timeout"> if your application requires it. ``` ``` lib/Mojo/Server/Prefork.pm 10 # No heartbeat (graceful stop) 9 $log->error("Worker $pid has no heartbeat ($ht seconds), restarting") and $w->{graceful} = $time 8 if !$w->{graceful} && ($w->{time} + $interval + $ht <= $time); 7 6 # Graceful stop with timeout 5 my $graceful = $w->{graceful} ||= $self->{graceful} ? $time : undef; 4 $log->info("Stopping worker $pid gracefully ($gt seconds)") and (kill 'QUIT', $pid or $self->_stopped($pid)) 3 if $graceful && !$w->{quit}++; ```