Project

General

Profile

tickets #123133

metrics.o.o: /var/log/messages flooded with GB of influxd messages

Added by cboltz 2 months ago. Updated 20 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Servers hosted in NBG
Target version:
-
Start date:
2023-01-14
Due date:
% Done:

0%

Estimated time:

Description

/var/log/messages on metrics.o.o gets flooded with gigabytes of influxd messages per day, with the result that the disk fills up within some hours.

I truncated /var/log/messages some minutes ago, but the disk will be full again in some hours - and I don't want to fight this windmill 3 times per day ;-)

Please fix the influxd config so that it does less logging.


Related issues

Related to openSUSE admin - tickets #123828: metrics.o.o: no OSRT:Review graphsNew2023-01-31

History

#1 Updated by cboltz 2 months ago

  • Private changed from Yes to No

#2 Updated by witekbedyk about 2 months ago

  • Assignee changed from witekbedyk to cboltz

The format of access logs has changed on 2022-12-23.
I have updated the regex to reflect that:
https://github.com/openSUSE/openSUSE-release-tools/pull/2920

Unfortunately access logs from the period 2022-12-23 to 2023-01-13 have been rotated already.

Is it possible to recover these logs from the backup to rerun the aggregation script on them?

#3 Updated by cboltz about 2 months ago

  • Assignee changed from cboltz to crameleon

I guess you mean the access_log from pontifex.i.o.o?

The only possibly helpful information I can give you is that there's an OLD_LOGS file on pontifex saying that old logfiles are stored on langley.suse.de. However, this file is from 2012, and I don't know if it's still true.

Georg, do you have a better answer? (Maybe the snapshots on the filer still have the access logs since Dec 23?)

#4 Updated by witekbedyk about 2 months ago

I found the logs at backup.infra.o.o in /backup/pontifex/var/log/apache2

I need the ones from download.opensuse.org and ipv6.download.opensuse.org.
I've seen the first missing entries are from 22.12.2022 already.

Could you please copy these logs somewhere that I can read them from metrics.infra.o.o?

#5 Updated by crameleon about 2 months ago

  • Assignee changed from crameleon to bmwiedemann

langley.suse.de.

I found some family and hiking photo albums from > 10 years ago on this machine, I assume those are not what you are looking for. But there are indeed some download.opensuse.org and ipv6.download.opensuse.org logs as well which seem to have ended around 2019 and 2017 respectively. The ominous "OLD_LOGFILES" text file can be found as well.

The snapshots cover the whole LUN, I'm not sure I can mount and browse them easily to check for these log files. Bernhard, can you?

#6 Updated by pjessen about 2 months ago

#7 Updated by witekbedyk 20 days ago

  • Status changed from New to In Progress
  • Assignee changed from bmwiedemann to witekbedyk

#8 Updated by witekbedyk 20 days ago

  • Status changed from In Progress to Resolved

All missing logs have been restored from backup.infa.o.o. Kudos an Bernhard.
I've reaggregated the metrics and all the data is visible now again.

Also available in: Atom PDF