Actions
tickets #135527
closedmatrix.i.o.o hookshot bridge failures and CPU spikes
Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
IRC and Matrix
Target version:
-
Start date:
2023-09-11
Due date:
% Done:
100%
Estimated time:
Description
Lately I've noticed that sometimes when I enter Element on my phone that animation of syncing never stops and rooms don't sync latest messages.
Today I logged in on matrix.i.o.o and checked the system logs, and this loop was all over the place:
Sep 11 14:04:13 matrix systemd[1]: hookshot.service: Main process exited, code=exited, status=1/FAILURE
Sep 11 14:04:13 matrix systemd[1]: hookshot.service: Failed with result 'exit-code'.
Sep 11 14:04:13 matrix systemd[1]: hookshot.service: Consumed 1.576s CPU time.
Sep 11 14:04:14 matrix systemd[1]: hookshot.service: Scheduled restart job, restart counter is at 260306.
Sep 11 14:04:14 matrix systemd[1]: Stopped Matrix Appservice hookshot.
Sep 11 14:04:14 matrix systemd[1]: hookshot.service: Consumed 1.576s CPU time.
Sep 11 14:04:14 matrix systemd[1]: Started Matrix Appservice hookshot.
Sep 11 14:04:15 matrix node[14486]: node:internal/modules/cjs/loader:1051
Sep 11 14:04:15 matrix node[14486]: throw err;
Sep 11 14:04:15 matrix node[14486]: ^
Sep 11 14:04:15 matrix node[14486]: Error: Cannot find module 'rss-parser'
Sep 11 14:04:15 matrix node[14486]: Require stack:
Sep 11 14:04:15 matrix node[14486]: - /data/var/lib/matrix-synapse/hookshot/lib/feeds/FeedReader.js
Sep 11 14:04:15 matrix node[14486]: - /data/var/lib/matrix-synapse/hookshot/lib/Bridge.js
Sep 11 14:04:15 matrix node[14486]: - /data/var/lib/matrix-synapse/hookshot/lib/App/BridgeApp.js
Sep 11 14:04:15 matrix node[14486]: at Module._resolveFilename (node:internal/modules/cjs/loader:1048:15)
Sep 11 14:04:15 matrix node[14486]: at Module._load (node:internal/modules/cjs/loader:901:27)
Sep 11 14:04:15 matrix node[14486]: at Module.require (node:internal/modules/cjs/loader:1115:19)
Sep 11 14:04:15 matrix node[14486]: at require (node:internal/modules/helpers:130:18)
Sep 11 14:04:15 matrix node[14486]: at Object.<anonymous> (/data/var/lib/matrix-synapse/hookshot/lib/feeds/FeedReader.js:11:38)
Sep 11 14:04:15 matrix node[14486]: at Module._compile (node:internal/modules/cjs/loader:1233:14)
Sep 11 14:04:15 matrix node[14486]: at Module._extensions..js (node:internal/modules/cjs/loader:1287:10)
Sep 11 14:04:15 matrix node[14486]: at Module.load (node:internal/modules/cjs/loader:1091:32)
Sep 11 14:04:15 matrix node[14486]: at Module._load (node:internal/modules/cjs/loader:938:12)
Sep 11 14:04:15 matrix node[14486]: at Module.require (node:internal/modules/cjs/loader:1115:19)
Sep 11 14:04:15 matrix node[14486]: at require (node:internal/modules/helpers:130:18)
Sep 11 14:04:15 matrix node[14486]: at Object.<anonymous> (/data/var/lib/matrix-synapse/hookshot/lib/Bridge.js:34:22)
Sep 11 14:04:15 matrix node[14486]: at Module._compile (node:internal/modules/cjs/loader:1233:14)
Sep 11 14:04:15 matrix node[14486]: at Module._extensions..js (node:internal/modules/cjs/loader:1287:10)
Sep 11 14:04:15 matrix node[14486]: at Module.load (node:internal/modules/cjs/loader:1091:32)
Sep 11 14:04:15 matrix node[14486]: at Module._load (node:internal/modules/cjs/loader:938:12) {
Sep 11 14:04:15 matrix node[14486]: code: 'MODULE_NOT_FOUND',
Sep 11 14:04:15 matrix node[14486]: requireStack: [
Sep 11 14:04:15 matrix node[14486]: '/data/var/lib/matrix-synapse/hookshot/lib/feeds/FeedReader.js',
Sep 11 14:04:15 matrix node[14486]: '/data/var/lib/matrix-synapse/hookshot/lib/Bridge.js',
Sep 11 14:04:15 matrix node[14486]: '/data/var/lib/matrix-synapse/hookshot/lib/App/BridgeApp.js'
Sep 11 14:04:15 matrix node[14486]: ]
Sep 11 14:04:15 matrix node[14486]: }
Sep 11 14:04:15 matrix node[14486]: Node.js v20.5.1
Sep 11 14:04:15 matrix systemd[1]: hookshot.service: Main process exited, code=exited, status=1/FAILURE
Sep 11 14:04:15 matrix systemd[1]: hookshot.service: Failed with result 'exit-code'.
Sep 11 14:04:15 matrix systemd[1]: hookshot.service: Consumed 1.515s CPU time.
Sep 11 14:04:16 matrix systemd[1]: Stopped Matrix Appservice hookshot.
Sep 11 14:04:16 matrix systemd[1]: hookshot.service: Consumed 1.515s CPU time.
The hookshot bridge was failing and retrying, failing and retrying, due to Error: Cannot find module 'rss-parser'. What are we using the hookshot bridge for?
The system load was ~3 ~3 ~3 (higher than seen in the attached image) before I stopped hookshot.service. And after observing Htop for a while the load average is still high -- more than 2, and less than 3.
Files
| Screenshot from 2023-09-11 11-17-54.png (33 KB) Screenshot from 2023-09-11 11-17-54.png | Htop - used resources | ||
| Screenshot from 2023-09-11 11-18-53.png (76.1 KB) Screenshot from 2023-09-11 11-18-53.png | Htop - top CPU eaters |
Actions