tickets #123133
closedmetrics.o.o: /var/log/messages flooded with GB of influxd messages
0%
Description
/var/log/messages on metrics.o.o gets flooded with gigabytes of influxd messages per day, with the result that the disk fills up within some hours.
I truncated /var/log/messages some minutes ago, but the disk will be full again in some hours - and I don't want to fight this windmill 3 times per day ;-)
Please fix the influxd config so that it does less logging.
Updated by witekbedyk almost 2 years ago
- Assignee changed from witekbedyk to cboltz
The format of access logs has changed on 2022-12-23.
I have updated the regex to reflect that:
https://github.com/openSUSE/openSUSE-release-tools/pull/2920
Unfortunately access logs from the period 2022-12-23 to 2023-01-13 have been rotated already.
Is it possible to recover these logs from the backup to rerun the aggregation script on them?
Updated by cboltz almost 2 years ago
- Assignee changed from cboltz to crameleon
I guess you mean the access_log from pontifex.i.o.o?
The only possibly helpful information I can give you is that there's an OLD_LOGS file on pontifex saying that old logfiles are stored on langley.suse.de. However, this file is from 2012, and I don't know if it's still true.
Georg, do you have a better answer? (Maybe the snapshots on the filer still have the access logs since Dec 23?)
Updated by witekbedyk almost 2 years ago
I found the logs at backup.infra.o.o in /backup/pontifex/var/log/apache2
I need the ones from download.opensuse.org and ipv6.download.opensuse.org.
I've seen the first missing entries are from 22.12.2022 already.
Could you please copy these logs somewhere that I can read them from metrics.infra.o.o?
Updated by crameleon almost 2 years ago
- Assignee changed from crameleon to bmwiedemann
langley.suse.de.
I found some family and hiking photo albums from > 10 years ago on this machine, I assume those are not what you are looking for. But there are indeed some download.opensuse.org and ipv6.download.opensuse.org logs as well which seem to have ended around 2019 and 2017 respectively. The ominous "OLD_LOGFILES" text file can be found as well.
The snapshots cover the whole LUN, I'm not sure I can mount and browse them easily to check for these log files. Bernhard, can you?
Updated by pjessen almost 2 years ago
- Related to tickets #123828: metrics.o.o: no OSRT:Review graphs added
Updated by witekbedyk almost 2 years ago
- Status changed from New to In Progress
- Assignee changed from bmwiedemann to witekbedyk
Updated by witekbedyk almost 2 years ago
- Status changed from In Progress to Resolved
All missing logs have been restored from backup.infa.o.o. Kudos an Bernhard.
I've reaggregated the metrics and all the data is visible now again.