QA » openQA Project » openQA Infrastructure

Looks like the relevant device is /dev/vdb so this is not about assets/results:

martchus@openqa:~> df -h / /srv/ /var/lib/openqa/share /var/lib/openqa/testresults
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/vda1        20G    8,1G   11G   44% /
/dev/vdb         80G     61G   20G   77% /srv
/dev/vdc        7,0T    5,6T  1,5T   80% /var/lib/openqa/share
/dev/vdd        5,5T    4,6T 1021G   82% /var/lib/openqa

In fact, it might be about the PostgreSQL database:

martchus@openqa:~> mount | grep /dev/vdb
/dev/vdb on /srv type xfs (rw,noatime,nodiratime,attr2,inode64,logbufs=8,logbsize=256k,noquota)
/dev/vdb on /var/lib/pgsql type xfs (rw,noatime,nodiratime,attr2,inode64,logbufs=8,logbsize=256k,noquota)

Yes, /srv is the filesystem where we store mostly the database but a bit of other data as well. Sorry if that was not clear. I have mentioned "postgres" only as a tag, not in the text of the description

It is really mostly the database:

--- /srv
   54,0 GiB [##########] /PSQL10
    6,2 GiB [#         ] /log                                                                                                                                                                                                                                                                                               
    1,3 GiB [          ]  homes.img
   23,1 MiB [          ] /salt
   10,0 MiB [          ] /pillar

We've already shared some figures within the chat and the indexes using most of the disk space. Here are the commands I've used to check this out: https://github.com/Martchus/openQA-helper#show-postgresql-table-sizes

Not sure whether we can easily improve/optimize this. We also likely can't just drop most of the indexes because they are actually there for a reason.

So I'd follow the last suggestion for now and asked infra to increase the size of /dev/vdb, e.g. to 100 GiB (so far we have 80 GiB).

So I'd follow the last suggestion for now and asked infra to increase the size of /dev/vdb, e.g. to 100 GiB (so far we have 80 GiB).

Good. I assume you did so with ticket. Can you please reference the ticket? And did you include osd-admins@suse.de in CC?

About AC3: Should I change the threshold in our monitoring from 90 % to 80 % or should I asked Infra to change the threshold in their monitoring from 80 % to 90 %. Or we just keep the 2 alerts differently so the Infra alert serves as initial alert and our own alert as a last reminder.

I suggest to keep the 2 alerts but the other way around: Our alert first on 80% and the EngInfra alert on 90%, i.e. ask 'em to bump to 90%.

I've asked them: https://infra.nue.suse.com/SelfService/Display.html?id=186868