Project

General

Profile

Actions
Copy link

action #125306

closed

rebootmgr/D-Bus-related problem on backup.qa.suse.de

Added by mkittler over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
mkittler
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-03-02
Due date:
% Done:

0%

Estimated time:
Tags:
alert, infra

Description

Observation

          ID: rebootmgr.service
    Function: service.running
      Result: False
     Comment: Job for rebootmgr.service failed because the control process exited with error code.
              See "systemctl status rebootmgr.service" and "journalctl -xeu rebootmgr.service" for details.
     Started: 11:44:25.029939
    Duration: 237.716 ms
     Changes:
…
Summary for backup.qa.suse.de
--------------
Succeeded: 232 (changed=1)
Failed:      1

martchus@backup-vm:~> sudo journalctl --since '1 hour ago' -u rebootmgr.service
Mar 02 14:38:38 backup-vm systemd[1]: Starting Reboot Manager...
Mar 02 14:38:38 backup-vm rebootmgrd[2911]: Connection to D-BUS system message bus failed: The maximum number of active connections for UID 0 has been reached.
Mar 02 14:38:38 backup-vm systemd[1]: rebootmgr.service: Main process exited, code=exited, status=1/FAILURE
Mar 02 14:38:38 backup-vm systemd[1]: rebootmgr.service: Failed with result 'exit-code'.
Mar 02 14:38:38 backup-vm systemd[1]: rebootmgr.service: Unit process 1258 (rebootmgrd) remains running after unit stopped.
Mar 02 14:38:38 backup-vm systemd[1]: rebootmgr.service: Unit process 27964 (rebootmgrd) remains running after unit stopped.
Actions
Copy link
 #1

Updated by mkittler over 1 year ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by mkittler over 1 year ago

Despite rebootmgr.service being failed, systemd showed tons of processes for that service. Somehow this service turned into a fork bomb that until Connection to D-BUS system message bus failed: The maximum number of active connections for UID 0 has been reached.. Invoking sudo systemctl kill rebootmgr.service and sudo systemctl start rebootmgr.service fixed the issue. The unit is now active with only one sub process. Maybe this is worth reporting an upstream issue. I don't think there's a misconfiguration on our side but I also don't know this rebootmgr machinery very well.

Actions
Copy link
 #3

Updated by mkittler over 1 year ago

  • Subject changed from D-Bus-related problem on backup.qa.suse.de to rebootmgr/D-Bus-related problem on backup.qa.suse.de
Actions
Copy link
 #4

Updated by mkittler over 1 year ago

  • Status changed from In Progress to Feedback

For now I've filed https://github.com/SUSE/rebootmgr/issues/15 but since this issue is hard to reproduce I'm not sure whether the issue can be fixed any time soon. Not sure what to do about it on our side. Maybe we can just hope that we won't run into the problem again.

Actions
Copy link
 #5

Updated by okurz over 1 year ago

  • Target version set to Ready
Actions
Copy link
 #6

Updated by mkittler over 1 year ago

  • Status changed from Feedback to Resolved

The upstream bug report has been closed. Likely there's nothing one can do about it upstream anyways.

I've checked the dbus logs but could only find Mar 02 09:48:15 backup-vm dbus-daemon[717]: [system] The maximum number of active connections for UID 0 has been reached (max_connections_per_user=256) which is already a symptom.

Since this problem occurred only one time so far I wouldn't try to investigate this any further.

Actions
Copy link

Also available in: Atom PDF