coordination #80142: [saga][epic] Scale out openQA: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
Install Pgpool-II or PgBouncer before PostgreSQL for openQA instances, e.g. to be used on OSD
We've recently seen quite a few log messages like this.
"Aug 07 02:19:14 openqa openqa: DBI connect('dbname=openqa','geekotest',...) failed: FATAL: remaining connection slots are reserved for non-replication superuser connections at /usr/lib/perl5/vendor_perl/5.18.2/Mojo/Pg.pm line 83."
And there are no obvious connection leaks in the webui. So it appears that at peak times we may be reaching the
default PostgreSQL connection limit with gru jobs (100 minus a few reserved connections). But even when we are
not at peak i count at least 68 active connections on OSD. Most are idle, and that is a lot of wasted resources.
PostgreSQL operates like an old Apache prefork server, so every idle connection requires a forked process.
Pgpool-II (and the more minimalistic PgBouncer)
are little daemons that run like a reverse proxy in front of PostgreSQL and that require almost no resources
keeping even thousands of idle connections alive. Real PostgreSQL connections would only be used when
there are active queries, and they would be shared.
#1 Updated by okurz over 1 year ago
so you are suggesting this to be installed but so far there is no reference to it within the packages or so, shouldn't we try to handle that upstream to e.g. for example "recommend" this?
Also, I have seen the message you gave as well. However shouldn't we rather investigate why this suddenly recently appeared and not "jump to conclusions"?
- Project changed from openQA Infrastructure to openQA Project
- Subject changed from Install Pgpool-II or PgBouncer before PostgreSQL for OSD to Install Pgpool-II or PgBouncer before PostgreSQL for openQA instances, e.g. to be used on OSD
- Category set to Feature requests
- Parent task set to #80142