action #96713
closed
Slow grep in openqa-label-known-issues leads to high CPU usage
Added by tinita over 2 years ago.
Updated over 2 years ago.
Description
Observation¶
On 2021-08-10 we experienced problems on osd like a high load, and one of the reasons seemed to be slow grep commands called by openqa-label-known-issues.
For example this regex seemed to be problematic:
grep -qPzo (?s)2021-.*T.*Error connecting to <root@s390p.*.suse.de>: No route to host /tmp/tmp.gy3iDKQDrV
Running over 4 minutes sometimes. (Removing the (?s) made it run in only a few milliseconds on a file with about 500kb.)
We are now lowering the timeout for the post fail hook
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/538
But that will mean that the whole process can timeout because of one bad regex.
Suggestions¶
- Study current grep logic in openqa-label-known-issues
- The mentioned regex comes from this issue: #93119
- Maybe the regex can be improved
- Add a
timeout to the grep call
- Make sure we are informed in some way of timed out grep calls so we can improve the regexes
- Reduce
limit=200 (maximum number of jobs to check)
- Copied from action #96710: Error `Can't call method "write" on an undefined value` shows up in worker log leading to incompletes added
- Description updated (diff)
Add a timeout to the grep call
Thinking out loud here. The script has some hard-coded regexes where grep is called individually, but results from progress are passed in one go. So I assume to allow a timeout per title we'd have to add the overhead of calling grep every single time.
It might be more beneficial to join other grep calls and speed up the whole thing?
cdywan wrote:
Thinking out loud here. The script has some hard-coded regexes where grep is called individually, but results from progress are passed in one go.
Not sure what you mean.
We currently call grep for every title.
- Status changed from New to In Progress
- Assignee set to tinita
- Status changed from In Progress to Feedback
PR was merged, scripts on osd updated.
I'll watch the logs.
- Target version set to Ready
- Related to action #96807: Web UI is slow and Apache Response Time alert got triggered added
- Due date set to 2021-09-03
- Priority changed from Urgent to High
- Assignee changed from tinita to okurz
- Priority changed from High to Normal
- Due date changed from 2021-09-03 to 2021-09-10
- Status changed from Feedback to In Progress
Since the MR was merged I don't see problems. On OSD I could confirm that sometimes there are grep processes running with nice level 19 as expected. Monitoring in htop I see that telegraf pops up often but only with nice-level of 10. Mostly it's "openqa prefork" processes so that's fine. I don't think my change had negative impact. However we see that we book all 10 cores often so we should ask for increase in number of cores. Also we see alerts on CPU load lately. We can consider bumping the CPU load alert limits a bit as well.
- Copied to action #97943: Increase number of CPU cores on OSD VM due to high usage size:S added
- Status changed from In Progress to Feedback
- Status changed from Feedback to Resolved
Also available in: Atom
PDF