openSUSE admin

It appears the issue is caused by a full disk.

/dev/vdb                         200G  200G   20K 100% /var/log

The logs have not been moved to langley since the network split for lack of infra providing a mechanism by which to send the data across network or a suitable place to back them up.

jberry wrote:

The logs have not been moved to langley since the network split for lack of infra providing a mechanism by which to send the data across network
or a suitable place to back them up.

Thanks. We don't have the room to archive logs on pontifex, so old stuff has to go. The 2018/2019 apache logs for 'download.o.o' take up about 2/3 of the 200Gb we currently have - I've cleared out the 2017 logs, but I expect I'll have to start on 2018 before long, we're 89% full. I guess we could expand the filesystem, if needed, but that'll only prolong the situation.

Unsurprisingly, /var/log/ on pontifex was full again (since two days, if I get the monitoring right).

I just deleted /var/log/nginx/downloadcontent/2017/ which had about 20 GB of logs - but once more, that's only a temporary solution. We really need to decide what to do with all the old logs!

cboltz wrote:

Unsurprisingly, /var/log/ on pontifex was full again (since two days, if I get the monitoring right).

Might this have to do with the database being unavailable, since Friday morning at least.
A normal month is about 10Gb, but for June 2019 sofar, we already have 14Gb.

I just deleted /var/log/nginx/downloadcontent/2017/ which had about 20 GB of logs - but once more,
that's only a temporary solution. We really need to decide what to do with all the old logs!

I think whoever has a need for them might want to provide some space of us to offload them?

Some of the logs in /var/log/apache2/2019/06 were not compressed, presumably because of the filesystem being full.

We are currently at 88% full, leaving about 26Gb available. Considering that we need about 5Gb working space when compressing, that is room for less than 2 months, i.e. for June and July, maybe a bit more. We could use /tmp for working space though. Maybe I'll check that out.

145G download.opensuse.org/
19G ipv6.download.opensuse.org/

Of the above, error_logs take up 4.2Gb.
Of the above, logs older than 1 year take up 30Gb.

If we were to keep logs for 1 year only, we should have a working set of 150Gb, that doesn't seem unreasonable.

The apache logs are growing rapidly, I expect due to the database problem (mirrordb1 being full). This afternoon we were at 86% full, now we have hit 92%.
I have cleared out apache2 logs more than 1 year old, now 77% full. That should keep us going until tomorrow morning.

I think whoever has a need for them might want to provide some space of us to offload them?

This is/was langley.suse.de, but since the network split no solution for migrating data across networks has been made available. As such, clearing the logs older than some date/size to keep a rotation seems like the best solution.