QA (public) » openQA Project (public)

Restarting the websocket server and livehandler will still cause 5xx responses so if that happened simultaneously it could explain 5xx responses. Those should be filtered out in our altering, though. (But maybe only on alert level and not in the graph.)

I'm currently checking logs via sudo journalctl -u openqa-webui.service | grep Reload but that command takes very long to execute. So far the logs don't show any unexpected behavior, though.

EDIT: It looks like reloads aren't happening that often:

martchus@openqa:~> sudo journalctl -u openqa-webui.service | grep Reload
Apr 23 21:04:54 openqa systemd[1]: Reloading The openQA web UI...
Apr 23 21:04:55 openqa systemd[1]: Reloaded The openQA web UI.
Apr 25 09:30:11 openqa systemd[1]: Reloading The openQA web UI...
Apr 25 09:30:12 openqa systemd[1]: Reloaded The openQA web UI.
May 02 07:19:36 openqa systemd[1]: Reloading The openQA web UI...
May 02 07:19:36 openqa systemd[1]: Reloaded The openQA web UI.
May 02 11:53:10 openqa systemd[1]: Reloading The openQA web UI... # probably triggered by #181433#note-6
May 02 11:53:10 openqa systemd[1]: Reloaded The openQA web UI.
May 02 12:18:16 openqa systemd[1]: Reloading The openQA web UI...
May 02 12:18:16 openqa systemd[1]: Reloaded The openQA web UI.
May 03 05:28:49 openqa systemd[1]: Reloading The openQA web UI...
May 03 05:28:50 openqa systemd[1]: Reloaded The openQA web UI.

Detailed logs for #181433#note-6:

May 02 11:53:10 openqa systemd[1]: Reloading The openQA web UI...
May 02 11:53:10 openqa systemd[1]: Reloaded The openQA web UI.
May 02 11:53:13 openqa openqa-webui-daemon[26241]: [INFO] Reading openqa config from: /etc/openqa/openqa.ini.d/openqa-salt.ini /etc/openqa/openqa.ini.d/xxx-increase-online-worker-limit.ini
May 02 11:53:13 openqa openqa-webui-daemon[26241]: [info] Loading external plugin AMQP
May 02 11:53:13 openqa openqa-webui-daemon[26241]: [info] Loading external plugin ObsRsync
May 02 11:53:15 openqa openqa-webui-daemon[26241]: [info] Listening at "http://127.0.0.1:9526?reuse=1"
May 02 11:53:15 openqa openqa-webui-daemon[26241]: Web application available at http://127.0.0.1:9526
May 02 11:53:15 openqa openqa-webui-daemon[26241]: [info] Listening at "http://[::1]:9526?reuse=1"
May 02 11:53:15 openqa openqa-webui-daemon[26241]: Web application available at http://[::1]:9526
May 02 11:53:15 openqa openqa-webui-daemon[26241]: [info] Manager 26241 started
May 02 11:53:15 openqa openqa-webui-daemon[26311]: [info] Worker 26311 started
May 02 11:53:15 openqa openqa-webui-daemon[26312]: [info] Worker 26312 started
May 02 11:53:15 openqa openqa-webui-daemon[26313]: [info] Worker 26313 started
May 02 11:53:15 openqa openqa-webui-daemon[26314]: [info] Worker 26314 started
May 02 11:53:15 openqa openqa-webui-daemon[26315]: [info] Worker 26315 started
May 02 11:53:15 openqa openqa-webui-daemon[26316]: [info] Worker 26316 started
May 02 11:53:15 openqa openqa-webui-daemon[26317]: [info] Worker 26317 started
May 02 11:53:15 openqa openqa-webui-daemon[26318]: [info] Worker 26318 started
May 02 11:53:15 openqa openqa-webui-daemon[26319]: [info] Worker 26319 started
May 02 11:53:15 openqa openqa-webui-daemon[26320]: [info] Worker 26320 started
May 02 11:53:15 openqa openqa-webui-daemon[26321]: [info] Worker 26321 started
May 02 11:53:15 openqa openqa-webui-daemon[26323]: [info] Worker 26323 started
May 02 11:53:15 openqa openqa-webui-daemon[26324]: [info] Worker 26324 started
May 02 11:53:15 openqa openqa-webui-daemon[26325]: [info] Worker 26325 started
May 02 11:53:15 openqa openqa-webui-daemon[26326]: [info] Worker 26326 started
May 02 11:53:15 openqa openqa-webui-daemon[26327]: [info] Worker 26327 started
May 02 11:53:15 openqa openqa-webui-daemon[26328]: [info] Worker 26328 started
May 02 11:53:15 openqa openqa-webui-daemon[26329]: [info] Worker 26329 started
May 02 11:53:15 openqa openqa-webui-daemon[26330]: [info] Worker 26330 started
May 02 11:53:15 openqa openqa-webui-daemon[26331]: [info] Worker 26331 started
May 02 11:53:15 openqa openqa-webui-daemon[26332]: [info] Worker 26332 started
May 02 11:53:15 openqa openqa-webui-daemon[26333]: [info] Worker 26333 started
May 02 11:53:15 openqa openqa-webui-daemon[26334]: [info] Worker 26334 started
May 02 11:53:15 openqa openqa-webui-daemon[26335]: [info] Worker 26335 started
May 02 11:53:15 openqa openqa-webui-daemon[26336]: [info] Worker 26336 started
May 02 11:53:15 openqa openqa-webui-daemon[26337]: [info] Worker 26337 started
May 02 11:53:15 openqa openqa-webui-daemon[26338]: [info] Worker 26338 started
May 02 11:53:15 openqa openqa-webui-daemon[26339]: [info] Worker 26339 started
May 02 11:53:15 openqa openqa-webui-daemon[26340]: [info] Worker 26340 started
May 02 11:53:15 openqa openqa-webui-daemon[26241]: [info] Creating process id file "/var/lib/openqa/webui/prefork-1.pid"
May 02 11:53:15 openqa openqa-webui-daemon[26341]: [info] Worker 26341 started
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 3438 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 7924 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 7391 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 8052 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 6066 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 8522 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 7313 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 10929 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 9487 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 13501 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 8321 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 6952 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 2762 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 12032 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 17420 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 5170 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 1744 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 4320 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 3308 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 3522 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 2552 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 10726 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 7832 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 5956 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 4241 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 7307 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 11029 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 17766 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 23750 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Stopping worker 30253 gracefully (800 seconds)
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 5956 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 8321 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 3522 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 12032 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 7832 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 8522 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 17420 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 10929 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 17766 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 7391 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 9487 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 3438 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 10726 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 7313 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 4241 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 23750 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 11029 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 3308 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 7307 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 1744 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 2762 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 7924 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 5170 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 8052 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 4320 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 2552 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 6066 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 30253 stopped
May 02 11:53:15 openqa openqa-webui-daemon[6381]: [info] Worker 6952 stopped
May 02 11:53:17 openqa openqa-webui-daemon[6381]: [info] Worker 13501 stopped
May 02 11:53:17 openqa openqa-webui-daemon[6381]: [info] Manager 6381 stopped

In /var/log/nginx/error.log-20250501.xz I see a bunch of Connection reset by peer messages that might be related as they might have ended up in a 5xx reply from NGINX and started only 2 seconds after the reload was triggered:

# first of such messages
2025/05/02 11:53:17 [error] 15748#15748: *259698987 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a07:de40:b203:12:262:bff:fe31:b160, server: openqa.suse.de, request: "POST /api/v1/jobs/17539491/artefact HTTP/1.1", upstream: "http://[::1]:9526/api/v1/jobs/17539491/artefact", host: "openqa.suse.de"
…
2025/05/02 12:19:11 [error] 15748#15748: *259940185 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a07:de40:b230:1:42f2:e9ff:fe31:5460, server: openqa.suse.de, request: "POST /api/v1/jobs/17539950/status HTTP/1.1", upstream: "http://[::1]:9526/api/v1/jobs/17539950/status", host: "openqa.suse.de"
2025/05/02 12:19:11 [error] 15748#15748: *259940196 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a07:de40:b203:12:262:bff:fe31:6430, server: openqa.suse.de, request: "POST /api/v1/jobs/17539589/status HTTP/1.1", upstream: "http://[::1]:9526/api/v1/jobs/17539589/status", host: "openqa.suse.de"
2025/05/02 12:19:11 [error] 15748#15748: *259940201 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a07:de40:b203:12:262:bff:fe31:6430, server: openqa.suse.de, request: "POST /api/v1/jobs/17540606/status HTTP/1.1", upstream: "http://[::1]:9526/api/v1/jobs/17540606/status", host: "openqa.suse.de"
2025/05/02 12:19:11 [error] 15748#15748: *259940208 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a07:de40:b203:12:7ec2:55ff:fe24:de70, server: openqa.suse.de, request: "POST /api/v1/jobs/17539662/status HTTP/1.1", upstream: "http://[::1]:9526/api/v1/jobs/17539662/status", host: "openqa.suse.de"
2025/05/02 12:19:11 [error] 15748#15748: *259940215 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a07:de40:b203:12:262:bff:fe31:6430, server: openqa.suse.de, request: "POST /api/v1/jobs/17539661/status HTTP/1.1", upstream: "http://[::1]:9526/api/v1/jobs/17539661/status", host: "openqa.suse.de"
2025/05/02 12:19:11 [error] 15748#15748: *259940230 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a07:de40:b203:12:7ec2:55ff:fe24:de9c, server: openqa.suse.de, request: "GET /api/v1/whoami HTTP/1.1", upstream: "http://[::1]:9526/api/v1/whoami", host: "openqa.suse.de"

…
# last of such message
2025/05/02 12:19:11 [error] 15748#15748: *259940626 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a07:de40:b203:12:7ec2:55ff:fe24:dc34, server: openqa.suse.de, request: "GET /tests/17535800/asset/hdd/publiccloud_tools_0074.qcow2 HTTP/1.1", upstream: "http://[::1]:9526/tests/17535800/asset/hdd/publiccloud_tools_0074.qcow2", host: "openqa.suse.de"

The logs show that all Mojolicious workers a granted a grace period of 800 seconds to stop. However, it only took 2 seconds (from 11:53:15 to 11:53:17). So Minion workers being forcefully stopped after exceeding the grace period is not the cause. After those 2 seconds the "manager" process is stopped as well. Only then we start to see error messaged on the NGINX side. And those are not "Connection refused" but "Connection reset by peer". So perhaps these connections have already been established but haven't been processed by any worker. After all workers have been stopped the management process is also stopped and connections that were still in the queue but have never been assigned to a worker are just dropped. This is of course just a theory; it would make more sense if the management process would still process all requests and dropping the queue like this is quite unexpected for a graceful shutdown.

I guess the problem could be reduced by introducing a delay between starting the new service and stopping the old one (so the old one can still work down pending requests). Of course adding a delay is not nice. We already stop the service gracefully via SIGQUIT. Maybe the behavior could be fixed upstream.

Note that there are also messages starting when the other reload was triggered that day:

2025/05/02 07:19:47 [error] 15748#15748: *253725755 connect() failed (111: Connection refused) while connecting to upstream, client: 2a07:de40:b203:12:7ec2:55ff:fe24:dc4c, server: openqa.suse.de, request: "GET /api/v1/ws/2846 HTTP/1.1", upstream: "http://[::1]:9527/api/v1/ws/2846", host: "openqa.suse.de"

However, those were exclusively about the ws route and as mentioned before the websocket server is still expected to produce these errors which in turn are expected to be excluded from alerting.

This can also be reproduced locally by running script/openqa-webui-daemon in one terminal and while true; do curl -s -o /dev/null -w "%{http_code}\n" http://localhost | grep -v 200; done in another. When sending SIGHUP to the "daemon" script (which is actually starting a preforking server) one gets sometimes a 502 response.

Then I tested with MOJO_SERVER_DEBUG=1 and while true; do code=$(curl -s -o /dev/null -w "%{http_code}\n" http://localhost); [[ $code != 200 ]] && echo "$code" && break; done to avoid too many log messages. I also checked sudo tail -f /var/log/nginx/error.log at the same time.

On the NGINX side I got the same error message as we see in production:

2025/05/07 12:15:19 [error] 14648#14648: *6571 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: ::1, server: openqa.suse.de, request: "GET / HTTP/1.1", upstream: "http://[::1]:9526/", host: "localhost"

Introducing a delay of 5 seconds before sending SIGQUIT to the old prefork service is not preventing this problem. That makes sense after having a closer look at the code but means we cannot work around the issue that simple.

Probably NGINX uses a sequence of connect() and recv() to connect to the Mojolicious app receiving a response. The connect() call still succeeds but then the Mojolicious app terminates and the subsequent recv() fails therefore with Connection reset by peer.

The last message from the Mojo app via MOJO_SERVER_DEBUG=1 is:

-- Accept d84e9620758fbb3af15aa0910180504c (::1)
-- Server <<< Client ()
GET / HTTP/1.0\x0d
Host: localhost\x0d
X-Forwarded-Host: localhost:80\x0d
X-Forwarded-Server: localhost\x0d
X-Forwarded-For: ::1\x0d
X-Forwarded-Proto: http\x0d
Connection: close\x0d
User-Agent: curl/8.13.0\x0d
Accept: */*\x0d
\x0d

[trace] [HcgN9flzpe-Z] GET "/"
[trace] [HcgN9flzpe-Z] Routing to a callback
[trace] [HcgN9flzpe-Z] Rendering cached template "main/index.html.ep"
[trace] [HcgN9flzpe-Z] Rendering cached template "layouts/info.html.ep"
[trace] [HcgN9flzpe-Z] Rendering cached template "layouts/flash_messages.html.ep"
[trace] [HcgN9flzpe-Z] Rendering cached template "branding/openqa.suse.de/docbox.html.ep"
[trace] [HcgN9flzpe-Z] Rendering cached template "branding/openqa.suse.de/sponsorbox.html.ep"
[trace] [HcgN9flzpe-Z] Rendering cached template "layouts/bootstrap.html.ep"
[trace] [HcgN9flzpe-Z] Rendering cached template "layouts/navbar.html.ep"
[trace] [HcgN9flzpe-Z] Template "branding/openqa.suse.de/links_footer_left.html.ep" not found
[trace] [HcgN9flzpe-Z] Template "branding/plain/links_footer_left.html.ep" not found
[trace] [HcgN9flzpe-Z] Template "branding/openqa.suse.de/links_footer_right.html.ep" not found
[trace] [HcgN9flzpe-Z] Template "branding/plain/links_footer_right.html.ep" not found
[trace] [HcgN9flzpe-Z] 200 OK (0.035941s, 27.823/s)
-- Server >>> Client (http://localhost/)
HTTP/1.1 200 OK\x0d
Content-Length: 28651\x0d
Content-Type: text/html;charset=UTF-8\x0d
Date: Wed, 07 May 2025 10:24:50 GMT\x0d
Server: Mojolicious (Perl)\x0d
Set-Cookie: mojolicious=eyJjc3JmX3Rva2VuIjoiNzEyNzQwZDM3NzJhMjc0ZTQ3YzZmMWM3N2IwOWEwNjJiYjkzZDU2ZiIsImV4cGlyZXMiOjE3NDY3ODYyOTB9--c54346300175891486e2864dbcf271073ef6fcbd9648a43f56ba9e2fe480c69a; expires=Fri, 09 May 2025 10:24:50 GMT; path=/; HttpOnly; SameSite=Lax\x0d
Strict-Transport-Security: max-age=31536000; includeSubDomains\x0d
Vary: Accept-Encoding\x0d
\x0d
<!DOCTYPE html>
<html lang="en">
…
</html>

-- Server >>> Client (http://localhost/)

[info] Stopping worker 22472 gracefully (800 seconds)
[info] Stopping worker 22492 gracefully (800 seconds)
[info] Stopping worker 22482 gracefully (800 seconds)
[info] Stopping worker 22469 gracefully (800 seconds)
[info] Stopping worker 22488 gracefully (800 seconds)
[info] Stopping worker 22477 gracefully (800 seconds)
[info] Stopping worker 22485 gracefully (800 seconds)
[info] Stopping worker 22497 gracefully (800 seconds)
[info] Stopping worker 22475 gracefully (800 seconds)
[info] Stopping worker 22487 gracefully (800 seconds)
[info] Stopping worker 22495 gracefully (800 seconds)
[info] Stopping worker 22478 gracefully (800 seconds)
[info] Stopping worker 22486 gracefully (800 seconds)
[info] Stopping worker 22476 gracefully (800 seconds)
[info] Stopping worker 22496 gracefully (800 seconds)
[info] Stopping worker 22468 gracefully (800 seconds)
[info] Stopping worker 22480 gracefully (800 seconds)
[info] Stopping worker 22483 gracefully (800 seconds)
[info] Stopping worker 22484 gracefully (800 seconds)
[info] Stopping worker 22489 gracefully (800 seconds)
[info] Stopping worker 22471 gracefully (800 seconds)
[info] Stopping worker 22491 gracefully (800 seconds)
[info] Stopping worker 22479 gracefully (800 seconds)
[info] Stopping worker 22481 gracefully (800 seconds)
[info] Stopping worker 22490 gracefully (800 seconds)
[info] Stopping worker 22494 gracefully (800 seconds)
[info] Stopping worker 22493 gracefully (800 seconds)
[info] Stopping worker 22473 gracefully (800 seconds)
[info] Stopping worker 22470 gracefully (800 seconds)
[info] Stopping worker 22474 gracefully (800 seconds)
[info] Worker 22469 stopped
[info] Worker 22472 stopped
[info] Worker 22492 stopped
[info] Worker 22481 stopped
[info] Worker 22482 stopped
[info] Worker 22497 stopped
[info] Worker 22491 stopped
[info] Worker 22476 stopped
[info] Worker 22475 stopped
[info] Worker 22488 stopped
[info] Worker 22477 stopped
[info] Worker 22485 stopped
[info] Worker 22478 stopped
[info] Worker 22468 stopped
[info] Worker 22494 stopped
[info] Worker 22474 stopped
[info] Worker 22490 stopped
[info] Worker 22495 stopped
[info] Worker 22493 stopped
[info] Worker 22479 stopped
[info] Worker 22473 stopped
[info] Worker 22487 stopped
[info] Worker 22470 stopped
[info] Worker 22486 stopped
[info] Worker 22489 stopped
[info] Worker 22483 stopped
[info] Worker 22471 stopped
[info] Worker 22484 stopped
[info] Worker 22480 stopped
[info] Worker 22496 stopped
[info] Manager 22462 stopped

Before there is similar output:

…
-- Accept b04ff0ab0e8bb409a3109bf1dba9dc02 (::1)
-- Server <<< Client ()
GET / HTTP/1.0\x0d
Host: localhost\x0d
X-Forwarded-Host: localhost:80\x0d
X-Forwarded-Server: localhost\x0d
X-Forwarded-For: ::1\x0d
X-Forwarded-Proto: http\x0d
Connection: close\x0d
User-Agent: curl/8.13.0\x0d
Accept: */*\x0d
\x0d

[trace] [E_FNsuVVXjSf] GET "/"
[trace] [E_FNsuVVXjSf] Routing to a callback
…
[trace] [E_FNsuVVXjSf] 200 OK (0.037936s, 26.360/s)
-- Server >>> Client (http://localhost/)
HTTP/1.1 200 OK\x0d
Content-Length: 28651\x0d
Content-Type: text/html;charset=UTF-8\x0d
Date: Wed, 07 May 2025 10:24:50 GMT\x0d
Server: Mojolicious (Perl)\x0d
Set-Cookie: mojolicious=eyJjc3JmX3Rva2VuIjoiNzI5MmY5OTIzZDliZTQ3YzI2M2NiYzFiMzA0NjU3MzNmYWI1MDg1YSIsImV4cGlyZXMiOjE3NDY3ODYyOTB9--30a247f3afdc16b70b407846919e5487ee7f212815d0c72b1aa21c7a44c243b7; expires=Fri, 09 May 2025 10:24:50 GMT; path=/; HttpOnly; SameSite=Lax\x0d
Strict-Transport-Security: max-age=31536000; includeSubDomains\x0d
Vary: Accept-Encoding\x0d
\x0d
<!DOCTYPE html>
<html lang="en">
…
-- Accept 79a2761adc3dc66c5c3312f90e02b58f (::1)
-- Server <<< Client ()
GET / HTTP/1.0\x0d
Host: localhost\x0d
X-Forwarded-Host: localhost:80\x0d
X-Forwarded-Server: localhost\x0d
X-Forwarded-For: ::1\x0d
X-Forwarded-Proto: http\x0d
Connection: close\x0d
User-Agent: curl/8.13.0\x0d
Accept: */*\x0d
\x0d

[trace] [c_BVi5HuYXOj] GET "/"
[trace] [c_BVi5HuYXOj] Routing to a callback
…
[trace] [c_BVi5HuYXOj] 200 OK (0.035126s, 28.469/s)
-- Server >>> Client (http://localhost/)
HTTP/1.1 200 OK\x0d
Content-Length: 28651\x0d
Content-Type: text/html;charset=UTF-8\x0d
Date: Wed, 07 May 2025 10:24:50 GMT\x0d
Server: Mojolicious (Perl)\x0d
Set-Cookie: mojolicious=eyJjc3JmX3Rva2VuIjoiNzkyNTYwYjRlNDMzNjM5Njg4NmEyNTRjYmIxMmM4YWUzMTY4MzQ3OCIsImV4cGlyZXMiOjE3NDY3ODYyOTB9--e2eaa5e4400b7ac4099ddadf35fdfd57ebd9359b81f6c8a6b1af9495aefeba65; expires=Fri, 09 May 2025 10:24:50 GMT; path=/; HttpOnly; SameSite=Lax\x0d
Strict-Transport-Security: max-age=31536000; includeSubDomains\x0d
Vary: Accept-Encoding\x0d
\x0d
<!DOCTYPE html>
<html lang="en">
…