openSUSE admin

pjessen wrote:

Indeed, it looks like nginx stopped yesterday at 1045UTC.

Jan 20 10:24:02 mailman3 systemd[1]: Reloading The nginx HTTP and reverse proxy server...
Jan 20 10:24:02 mailman3 systemd[1]: Reloaded The nginx HTTP and reverse proxy server.
Jan 21 10:45:13 mailman3 systemd[1]: nginx.service: Main process exited, code=killed, status=9/KILL
Jan 21 10:45:13 mailman3 systemd[1]: nginx.service: Killing process 24437 (nginx) with signal SIGKILL.
Jan 21 10:45:13 mailman3 systemd[1]: nginx.service: Killing process 13820 (nginx) with signal SIGKILL.
Jan 21 10:45:13 mailman3 systemd[1]: nginx.service: Killing process 13873 (nginx) with signal SIGKILL.
Jan 21 10:45:13 mailman3 systemd[1]: nginx.service: Killing process 13897 (nginx) with signal SIGKILL.
Jan 21 10:45:13 mailman3 systemd[1]: nginx.service: Killing process 13919 (nginx) with signal SIGKILL.
Jan 21 10:45:14 mailman3 systemd[1]: nginx.service: Failed with result 'signal'.
Jan 21 10:45:14 mailman3 systemd[1]: nginx.service: Consumed 20h 13min 18.620s CPU time.

I have restarted nginx.

Seems to work again. Thx. Feel free to close it.

At first, dmesg did not show nginx being killed by the oom killer, but from /var/log/messages :

2023-01-21T10:45:13.488872+00:00 mailman3 kernel: [6224189.860720][ T9631] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global
_oom,task_memcg=/system.slice,task=nginx,pid=32317,uid=0
2023-01-21T10:45:13.488873+00:00 mailman3 kernel: [6224189.860757][ T9631] Out of memory: Killed process 32317 (nginx) total-vm:1384308kB, anon-rss:1346128kB
, file-rss:68kB, shmem-rss:4kB, UID:0 pgtables:2748kB oom_score_adj:0

FYI: https://lists.opensuse.org/ is down again.

Yes, I've been trying to restart nginx all day. It seems to be running now.

Cop-out: Because nginx seems to have become the preferred victim, I have added automatic nginx restart.

# /etc/systemd/system/nginx.service.d/restart.conf
[Service]
RestartSec=600s
Restart=on-failure

pjessen wrote:

Cop-out: Because nginx seems to have become the preferred victim, I have added automatic nginx restart.

# /etc/systemd/system/nginx.service.d/restart.conf
[Service]
RestartSec=600s
Restart=on-failure

Unfortunately, https://lists.opensuse.org/ is down again for quite some time now. By the way, why doesn't this issue show up at https://status.opensuse.org/?

fkrueger wrote:

Unfortunately, https://lists.opensuse.org/ is down again for quite some time now.

Yes, it looks like the automatic restart of nginx is working, but when the rest of the machine is misbehaving ...

By the way, why doesn't this issue show up at https://status.opensuse.org/?

Updating https://status.opensuse.org/ is not automatic, it is a manual operation.

pjessen wrote:

fkrueger wrote:

Unfortunately, https://lists.opensuse.org/ is down again for quite some time now.

Yes, it looks like the automatic restart of nginx is working, but when the rest of the machine is misbehaving ...

postfix also got oom killed, last night around 2220.