Project

General

Profile

tickets #125897

pagure01.infra.o.o / code-o-o - redis killed due to OOM, system RAM and swap maxed out

Added by lkocman 3 months ago. Updated 3 months ago.

Status:
Workable
Priority:
Normal
Assignee:
-
Category:
Git(lab|hub)/ Pagure
Target version:
-
Start date:
2023-03-13
Due date:
% Done:

0%

Estimated time:

Description

Hello team,

I tried to reject this particular issue https://code.opensuse.org/leap/features/issue/105
However, I always received Fatal Error (500)

Could you please look at it?

Thank you

Related issues

Related to openSUSE admin - tickets #125687: code.o.o error when creating new reposNew2023-03-09

History

#1 Updated by lkocman 3 months ago

I just figured out that the same 500 happens if I try to comment to the issue.

#2 Updated by lkocman 3 months ago

Logs mention that connection to redis failed.

#3 Updated by Pharaoh_Atem 3 months ago

This is what I'm seeing at a cursory glance...

pagure01 (pagure):~ # systemctl status redis@default
× redis@default.service - Redis instance: default
     Loaded: loaded (/usr/lib/systemd/system/redis@.service; enabled; vendor preset: disabled)
     Active: failed (Result: signal) since Wed 2023-03-08 16:38:42 UTC; 4 days ago
    Process: 25073 ExecStart=/usr/sbin/redis-server /etc/redis/default.conf (code=killed, signal=KILL)
   Main PID: 25073 (code=killed, signal=KILL)
     Status: "Redis is loading..."

Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Main process exited, code=killed, status=9/KILL
Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Failed with result 'signal'.
Mar 08 16:38:42 pagure01 systemd[1]: Failed to start Redis instance: default.
Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Scheduled restart job, restart counter is at 2183.
Mar 08 16:38:42 pagure01 systemd[1]: Stopped Redis instance: default.
Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Start request repeated too quickly.
Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Failed with result 'signal'.
Mar 08 16:38:42 pagure01 systemd[1]: Failed to start Redis instance: default.
pagure01 (pagure):~ # systemctl start redis@default
Job for redis@default.service failed because a fatal signal was delivered to the control process.
See "systemctl status redis@default.service" and "journalctl -xeu redis@default.service" for details.

#4 Updated by Pharaoh_Atem 3 months ago

Looks like redis is getting killed because we're out of memory. Trying to restart service to free up RAM.

#5 Updated by Pharaoh_Atem 3 months ago

  • Subject changed from code-o-o - unable to reject ticket to code-o-o - redis killed due to OOM, system RAM and swap maxed out

It looks like we're maxed out on RAM and swap, can we double the amount of RAM for the pagure VM?

#6 Updated by Pharaoh_Atem 3 months ago

  • Private changed from Yes to No

#7 Updated by Pharaoh_Atem 3 months ago

  • Subject changed from code-o-o - redis killed due to OOM, system RAM and swap maxed out to pagure01.infra.o.o / code-o-o - redis killed due to OOM, system RAM and swap maxed out

#9 Updated by crameleon 3 months ago

  • Status changed from New to In Progress
  • Assignee set to crameleon

#10 Updated by crameleon 3 months ago

  • Status changed from In Progress to Workable
  • Assignee deleted (crameleon)

Thanks for the SD ticket, Lubos.

Memory change is done.

I recommend the application owner to configure the maxmemory and maxmemory-policy options in Redis to mitigate such issues.

#11 Updated by pjessen 3 months ago

Also available in: Atom PDF