QA (public) » openQA Project (public)

I don't remember exactly how we envisioned this to work.

Did we have figures supporting the fact that the load of the websocket server and scheduler are actually high in the problematic situation? I'm asking that because I highly doubt that this is the case. We have already established that at least the websocket server does not cause much CPU load. The same is probably true for the scheduler. I also doubt that both cause a considerable amount of I/O load. Maybe the worker processes of PostgreSQL cause a high I/O load (or even CPU load) instead (which would make it hard to pin-down this load to the scheduling problem). Maybe none of the processes cause a high load because locks held by some txn are the bottleneck.

I would probably approach this by trying to solve the actual problem first - which means provoking it in some way locally with the help of some servers with enough RAM¹. Then I'd closely observe the resource usage locally and what causes this exactly. Depending on the findings fix the problem itself might be simpler than adding a dynamic limit in case the problem occurs². So adding a dynamic limit might not make sense and adding it blindly without knowing what to look for even less.

So I guess by trying to fight the symptoms first we're approaching the problem from the wrong angle.

¹ When working on this before I noticed that with "only" 32 GiB RAM the number of worker slots I could start on my laptop was quite limited.
² Adding a dynamic limit isn't that trivial.

mkittler wrote in #note-8:

I would probably approach this by trying to solve the actual problem first - which means provoking it in some way locally with the help of some servers with enough RAM¹. Then I'd closely observe the resource usage locally and what causes this exactly.

I agree. That is how I interpret the suggestion "Come up with a definition of the critical websocket+scheduler load based on 'overload experiments'"