Added by livdywan over 2 years ago. Updated over 2 years ago.
0%
Description
telegraf is going up and downI saw the CPU alert trigger in the meantime, but I couldn't confirm a correlation with Telegraf's spikes which just seem to continue.
Switched it off temporarily for testing via sudo systemctl disable --now telegraf and I see openqa processes maxing out at 54% as the worst offenders now. SWitching it back on via enable the spikes are back fully.
Also checked the /debug/vars route although I couldn't see anything useful there and reverted the logwarn support which wasn't correctly configured but that also didn't affect CPU usage.
In the weekly it was suggested we could reduceincrease the interval which is currently 10s on the web UI and 1m on workers. We also have 0s jitter, which upstream docs recommend be used to spread out plugins for better performance. MR here: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/548
Setting due date based on mean cycle time of SUSE QE Tools
Another suggestion from the extended daily was to adjust the nice level via salt (maybe via Exec or salt states if possible), so I'll prepare an MR for that: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/551
merged. I did on osd systemctl daemon-reload && systemctl restart telegraf. I can confirm that telegraf runs with nice-level 10 now. What's next?
okurz wrote:
merged. I did on osd
systemctl daemon-reload && systemctl restart telegraf. I can confirm that telegraf runs with nice-level 10 now. What's next?
It looks like telegraf is on average using a lot less CPU than before, so I'm inclined to consider this a success.
cdywan wrote:
It looks like telegraf is on average using a lot less CPU than before, so I'm inclined to consider this a success.
Hence resolving.