tickets #127766
closedpontifex - cannot access '/var/log/messages*xz': No such file or directory
100%
Description
What is being doing with rotated logs ? /var/log/messages is supposed by to be kept for 30 days, but there are no compressed rotated logs. This happened today, on 16 March, 15 February, 14 January, 13 December, 13 November. Monthly?
Updated by crameleon about 1 year ago
- Category set to Servers hosted in NBG
- Private changed from Yes to No
Updated by pjessen about 1 year ago
- Status changed from New to In Progress
- Assignee set to pjessen
Found it. It is a monthly cron job being run out of /etc/cron.monthly ... I have never ever used those hourly/daily/weekly/monthly directories, dunno why it suddenly occurred to me to check.
We are running the script /root/bin/copy_logs_to_backup.sh, at the middle of the month, to copy all /var/log/*xz to "backup.infra.opensuse.org" and then delete them.
First thought - this is a good idea, just not quite right. It should be done every day, not every month, and the most recent logfiles should backed up, but not deleted. It seems like a perfect job for logrotate - ISTR some option for having something done to rotated-out logfiles. I don't see who wrote that script and it isn't packaged. Does anyone know who?
Updated by pjessen about 1 year ago
Mostly notes to self:
- the preremove script is given the full path of the logfile.
- have created a new rsync module on backup.i.o.o
- on pontifex, /var/log/messages and /var/log/mail that are being rotated out, are first copied to backup.i.o.o.
- because the current backup script does a find of all *xz files in /var/log/, I will probably amend it to skip /var/log/messages and /var/log/mail
Updated by pjessen about 1 year ago
Have now amended the log backup script /root/bin/copy_logs_to_backup.sh to ignore messages*.xz and mail*xz - those will be taken care of by a preremove script in the logrotate config. On backup.i.o.o I have created a separate directory /backup/logarchive with the intention of enabling all of our hosts to archive logs.
TODO - cleanup job on backup.i.o.o, to keep the amount of archived logs to something reasonable.
Updated by pjessen about 1 year ago
Slight surprise - the logrotate::preremove script is also called when a rotated log has been compressed, and the uncompressed logfile is being removed.
Updated by pjessen about 1 year ago
Hmm, I am having trouble getting the script to work as part of logrotate, whereas it works fine when invoked manually. systemd hardening again? or apparmor? Hmm, I see no apparmor profile for logrotate.
Have moved the script to /usr/local/bin - let us see if that makes a difference.
Updated by crameleon about 1 year ago
May 08 00:00:01 pontifex2 systemd[1]: Starting Rotate log files...
May 08 00:00:01 pontifex2 logrotate[19046]: error: syslog:45 duplicate log entry for /var/log/mail
May 08 00:00:01 pontifex2 logrotate[19046]: error: found error in file syslog, skipping
May 08 00:07:10 pontifex2 logrotate[19046]: error: error running preremove script for /var/log/messages-20230508 of '/var/log/messages
May 08 00:07:10 pontifex2 logrotate[19046]: '. Not removing this file.
May 08 00:07:14 pontifex2 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
I commented out the /var/log/mail.info /var/log/mail.warn /var/log/mail.err line, now it is working fine:
May 09 16:33:33 pontifex2 systemd[1]: Starting Rotate log files...
May 09 16:38:47 pontifex2 systemd[1]: logrotate.service: Deactivated successfully.
May 09 16:38:47 pontifex2 systemd[1]: Finished Rotate log files.
Updated by pjessen about 1 year ago
crameleon wrote:
May 08 00:00:01 pontifex2 systemd[1]: Starting Rotate log files... May 08 00:00:01 pontifex2 logrotate[19046]: error: syslog:45 duplicate log entry for /var/log/mail May 08 00:00:01 pontifex2 logrotate[19046]: error: found error in file syslog, skipping May 08 00:07:10 pontifex2 logrotate[19046]: error: error running preremove script for /var/log/messages-20230508 of '/var/log/messages May 08 00:07:10 pontifex2 logrotate[19046]: '. Not removing this file. May 08 00:07:14 pontifex2 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILUREI commented out the
/var/log/mail.info /var/log/mail.warn /var/log/mail.errline, now it is working fine:
Hmm, I noticed that yesterday and just removed /var/log/mail from the line with the other three mail logs.
May 09 16:33:33 pontifex2 systemd[1]: Starting Rotate log files... May 09 16:38:47 pontifex2 systemd[1]: logrotate.service: Deactivated successfully. May 09 16:38:47 pontifex2 systemd[1]: Finished Rotate log files.
Well, the question is, did it execute any preremove steps and as I already did the archiving earlier there was probably nothing to do :-)
Updated by pjessen about 1 year ago
2023-05-10T00:02:20.783607+00:00 pontifex2 logrotate[22051]: error: error running preremove script for /var/log/messages-20230510 of '/var/log/messages
2023-05-10T00:02:20.783849+00:00 pontifex2 logrotate[22051]: '. Not removing this file.
2023-05-10T00:02:20.873620+00:00 pontifex2 logrotate[22051]: error: error running preremove script for /var/log/mail-20230510 of '/var/log/mail
2023-05-10T00:02:20.873789+00:00 pontifex2 logrotate[22051]: '. Not removing this file.
2023-05-10T00:02:24.206985+00:00 pontifex2 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
2023-05-10T00:02:24.207239+00:00 pontifex2 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Have moved the script to /usr/local/bin - let us see if that makes a difference.
Made no ddifference.
Updated by pjessen about 1 year ago
The correct preremove config is:
preremove
/usr/local/bin/archive_logfile.sh $1
endscript
Updated by pjessen about 1 year ago
Good stuff. Now I'll wait for a week or so, to see the first logfiles being rotated out.
Updated by pjessen about 1 year ago
- Due date set to 2023-05-19
- % Done changed from 0 to 50
Updated by pjessen about 1 year ago
- % Done changed from 50 to 80
It's working - today messages-20230418.xz was rotated out, and archived to backup.i.o.o. Cool.
Updated by pjessen about 1 year ago
- Status changed from In Progress to Resolved
- % Done changed from 80 to 100
This is both beautiful and elegant - even if I have to say so myself :-)
I'm going to think about making this a general procedure for archiving old logs.