Project

General

Profile

Actions

tickets #125897

open

pagure01.infra.o.o / code-o-o - redis killed due to OOM, system RAM and swap maxed out

Added by lkocman about 1 year ago. Updated 9 months ago.

Status:
Workable
Priority:
Normal
Assignee:
Category:
Git(lab|hub)
Target version:
-
Start date:
2023-03-13
Due date:
% Done:

0%

Estimated time:

Description

Hello team,

I tried to reject this particular issue https://code.opensuse.org/leap/features/issue/105
However, I always received Fatal Error (500)

Could you please look at it?

Thank you


Related issues 1 (1 open0 closed)

Related to openSUSE admin - tickets #125687: code.o.o error when creating new reposNew2023-03-09

Actions
Actions #1

Updated by lkocman about 1 year ago

I just figured out that the same 500 happens if I try to comment to the issue.

Actions #2

Updated by lkocman about 1 year ago

Logs mention that connection to redis failed.

Actions #3

Updated by Pharaoh_Atem about 1 year ago

This is what I'm seeing at a cursory glance...

pagure01 (pagure):~ # systemctl status redis@default
× redis@default.service - Redis instance: default
     Loaded: loaded (/usr/lib/systemd/system/redis@.service; enabled; vendor preset: disabled)
     Active: failed (Result: signal) since Wed 2023-03-08 16:38:42 UTC; 4 days ago
    Process: 25073 ExecStart=/usr/sbin/redis-server /etc/redis/default.conf (code=killed, signal=KILL)
   Main PID: 25073 (code=killed, signal=KILL)
     Status: "Redis is loading..."

Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Main process exited, code=killed, status=9/KILL
Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Failed with result 'signal'.
Mar 08 16:38:42 pagure01 systemd[1]: Failed to start Redis instance: default.
Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Scheduled restart job, restart counter is at 2183.
Mar 08 16:38:42 pagure01 systemd[1]: Stopped Redis instance: default.
Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Start request repeated too quickly.
Mar 08 16:38:42 pagure01 systemd[1]: redis@default.service: Failed with result 'signal'.
Mar 08 16:38:42 pagure01 systemd[1]: Failed to start Redis instance: default.
pagure01 (pagure):~ # systemctl start redis@default
Job for redis@default.service failed because a fatal signal was delivered to the control process.
See "systemctl status redis@default.service" and "journalctl -xeu redis@default.service" for details.

Actions #4

Updated by Pharaoh_Atem about 1 year ago

Looks like redis is getting killed because we're out of memory. Trying to restart service to free up RAM.

Actions #5

Updated by Pharaoh_Atem about 1 year ago

  • Subject changed from code-o-o - unable to reject ticket to code-o-o - redis killed due to OOM, system RAM and swap maxed out

It looks like we're maxed out on RAM and swap, can we double the amount of RAM for the pagure VM?

Actions #6

Updated by Pharaoh_Atem about 1 year ago

  • Private changed from Yes to No
Actions #7

Updated by Pharaoh_Atem about 1 year ago

  • Subject changed from code-o-o - redis killed due to OOM, system RAM and swap maxed out to pagure01.infra.o.o / code-o-o - redis killed due to OOM, system RAM and swap maxed out
Actions #9

Updated by crameleon about 1 year ago

  • Status changed from New to In Progress
  • Assignee set to crameleon
Actions #10

Updated by crameleon about 1 year ago

  • Status changed from In Progress to Workable
  • Assignee deleted (crameleon)

Thanks for the SD ticket, Lubos.

Memory change is done.

I recommend the application owner to configure the maxmemory and maxmemory-policy options in Redis to mitigate such issues.

Actions #11

Updated by pjessen about 1 year ago

Actions #12

Updated by crameleon 9 months ago

  • Assignee set to Pharaoh_Atem

Hi @Pharaoh_Atem, any comments on my suggestion?

Actions

Also available in: Atom PDF