tickets #52823
closednginx logs on pontifex - being rotated by /usr/sbin/nginx-compress-logs as well as by logrotate?
Added by pjessen almost 5 years ago. Updated 6 months ago.
50%
Description
logrotate runs every day at midnight, on logrotate.timer.
# cat /etc/logrotate.d/nginx
/var/log/nginx/*.log /var/log/nginx/*/*.log /var/log/nginx/*/*/*/*.log {
compress
dateext
maxage 365
rotate 99
size=+4096k
missingok
notifempty
create 644 root root
sharedscripts
postrotate
# -s reopen will use the pid file passed in the config file or the compiled in default path
/usr/sbin/nginx -s reopen
endscript
}
In addition, every day at 00:30, we also run /usr/sbin/nginx-compress-logs, a small script that does pretty much the same.
Also, as we are using cronologs with nginx, there is no need for logrotate to append year&date. (dateext).
Updated by pjessen almost 5 years ago
Uh, update - we're not using cronolog, it just looked like it - this is from /etc/nginx/vhosts.d/000-downloadcontent.conf
access_log /var/log/nginx/downloadcontent/$year/$month/$year-$month-$day-access.log main;
Something stop&starts nginx at midnight which gets a new log started, but I can't figure out what it is. I don't see any cronjob nor any systemd timer doing that.
Ah, it must be the "/usr/sbin/nginx -s reopen" from logrotate.
Logrotate does the compression but it never cleans out any old logs. I have disabled 'nginx-compress-logs' in /etc/cron.d/nginx-time-based-logfiles-helpers. I have also amended the logrotate config to not use "dateext" with /var/log/nginx/*/*/*/*.log
Updated by pjessen almost 5 years ago
- Due date set to 2019-06-11
- Status changed from New to In Progress
Updated by lrupp over 2 years ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
Hi there - and a Happy and Healthy 2022!
I'm currently closing old tickets which did not see much change.
If the main concern still exists and should be handled, please re-open by just replying to this Email.
Thanks in advance,
Lars
Updated by pjessen about 2 years ago
- Related to tickets #108197: pontifex - /usr/sbin/nginx-compress-logs vs /etc/logrotate.d/nginx added
Updated by pjessen about 2 years ago
- Status changed from Closed to New
- Assignee set to pjessen
- % Done changed from 100 to 50
I'll continue this here, rather than the new one I opened, #108197
Updated by pjessen about 2 years ago
In addition, every day at 00:30, we also run /usr/sbin/nginx-compress-logs, a small script that does pretty much the same.
The script also updates a symlink.
Occasionally, these errors are reported to admin-auto:
xz: /var/log/nginx/downloadcontent/2022/03/2022-03-09-access.log.xz: File exists
xz: /var/log/nginx/downloadcontent/2022/03/2022-03-12-access.log.xz: File exists
pontifex2 (download.o.o):~ # l /var/log/nginx/downloadcontent/2022/03/
total 158156
drwxr-xr-x 2 nginx nginx 4096 Mar 14 00:30 ./
drwxr-xr-x 5 nginx nginx 33 Mar 1 00:00 ../
-rw-r--r-- 1 nginx nginx 304 Mar 3 01:34 2022-03-02-access.log.xz
-rw-r--r-- 1 nginx nginx 4726920 Mar 6 00:03 2022-03-05-access.log.xz
-rw-r--r-- 1 nginx nginx 4284940 Mar 7 00:17 2022-03-06-access.log.xz
-rw-r--r-- 1 nginx nginx 10060608 Mar 8 00:13 2022-03-07-access.log.xz
-rw-r--r-- 1 nginx nginx 7297828 Mar 9 00:09 2022-03-08-access.log.xz
-rw-r--r-- 1 nginx nginx 339 Mar 10 00:45 2022-03-09-access.log
-rw-r--r-- 1 nginx nginx 12546652 Mar 10 00:09 2022-03-09-access.log.xz
-rw-r--r-- 1 nginx nginx 6964504 Mar 11 00:02 2022-03-10-access.log.xz
-rw-r--r-- 1 nginx nginx 8437036 Mar 12 00:00 2022-03-11-access.log.xz
-rw-r--r-- 1 nginx nginx 354 Mar 13 00:58 2022-03-12-access.log
-rw-r--r-- 1 nginx nginx 5173420 Mar 13 00:20 2022-03-12-access.log.xz
-rw-r--r-- 1 nginx nginx 5750752 Mar 14 00:02 2022-03-13-access.log.xz
2022-03-12-access.log contains one line, with an entry for 2022-03-13.
Updated by pjessen almost 2 years ago
- Subject changed from nginx logs on pontifex - being rotate by /usr/sbin/nginx-compress-logs as well as by logrotate? to nginx logs on pontifex - being rotated by /usr/sbin/nginx-compress-logs as well as by logrotate?
More unexpected errors from this process:
xz: /var/log/nginx/downloadcontent/2022/07/2022-07-20-access.log.xz: File exists
xz: /var/log/nginx/downloadcontent/2022/07/2022-07-22-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/downloadcontent/2022/07/2022-07-23-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/downloadcontent/2022/07/2022-07-25-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/downloadcontent/2022/07/2022-07-26-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/downloadcontent/2022/07/2022-07-19-access.log.xz: File exists
xz: /var/log/nginx/downloadcontent/2022/07/2022-07-21-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/downloadcontent/2022/07/2022-07-24-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/pontifex2/2022/07/2022-07-21-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/pontifex2/2022/07/2022-07-22-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/pontifex2/2022/07/2022-07-24-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/pontifex2/2022/07/2022-07-25-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/pontifex2/2022/07/2022-07-23-access.log.xz: Cannot set the file group: Operation not permitted
xz: /var/log/nginx/pontifex2/2022/07/2022-07-26-access.log.xz: Cannot set the file group: Operation not permitted
"Cannot set the file group: Operation not permitted" - apparmor?
Updated by cboltz almost 2 years ago
pjessen wrote:
"Cannot set the file group: Operation not permitted" - apparmor?
I don't see a profile that might be related to logrotation or xz, and audit.log doesn't say anything about it, so AppArmor is a bit ;-) unlikely.
Unfortunately, I don't know what else could cause the permission issue - even if script (and therefore xz) runs as user nginx, it has full permissions on those directories (directory and files owned by nginx:nginx).
(Hmm, maybe add a ls -al in the script before and after compressing the logs? That might give us a hint what's going on.)
.
One thing I noticed is that you disabled nginx-compress-logs in 2019(!), but your change was moved to a *.rpmsave :-/ (the new/active version has MAILTO added, so please don't just move the *.rpmsave back)
pontifex2 (download.o.o):/etc # head -n100 /etc/cron.d/nginx-time-based-logfiles-helpers*
==> /etc/cron.d/nginx-time-based-logfiles-helpers <==
MAILTO=admin-auto@opensuse.org
@monthly nginx /usr/sbin/nginx-logdirs
30 0 * * * nginx /usr/sbin/nginx-compress-logs
==> /etc/cron.d/nginx-time-based-logfiles-helpers.rpmsave <==
@monthly nginx /usr/sbin/nginx-logdirs
# 20190610 disabled, logrotate takes care of it. see https://progress.opensuse.org/issues/52823
#30 0 * * * nginx /usr/sbin/nginx-compress-logs
Updated by pjessen almost 2 years ago
cboltz wrote:
pjessen wrote:
"Cannot set the file group: Operation not permitted" - apparmor?
I don't see a profile that might be related to logrotation or xz, and audit.log doesn't say anything about it, so AppArmor is a bit ;-) unlikely.
Yeah, it was a wild guess. I did look at the log too and saw nothing.
One thing I noticed is that you disabled nginx-compress-logs in 2019(!), but your change was moved to a *.rpmsave :-/ (the new/active version has MAILTO added, so please don't just move the *.rpmsave back)
Got it. 2019, that was a lifetime ago ... thanks for reminding me.
I have disabled the cronjob again, maintaining the MAILTO. The "issue" of creating the symlink remains, but I think it is really minor. Maybe it could be worked into the logrotate config.
Updated by pjessen over 1 year ago
- Due date changed from 2019-06-11 to 2022-09-11
- Status changed from New to In Progress
Updated by pjessen over 1 year ago
- Due date changed from 2022-09-11 to 2022-08-23
- Category deleted (
Mirrors)
Because it's been three years, let me do a quick recap:
- the nginx package comes with its own logrotate config, which however looks only at /var/log/nginx/*.log (not being used).
- our nginx config uses
access_log /var/log/nginx/downloadcontent/$year/$month/$year-$month-$day-access.log - a new logfile is only opened due to logrotate doing
nginx -s reopenat midnight.
In essence - the package-supplied logrotate config does not really work because of our logging setup. It is however "needed" to do the reopen every day. I would much prefer a "clean" logrotate setup, but I am not keen to rework the nginx logging setup right now.
New scheme:
- do a log reopen at 00:00, by cronjob.
- log compression by script at 00:30, the previous cronjob.
- amend the logrotate config so it does nothing, and does not get overwritten on the next update.
- cronjob - will that get overwritten on the next update?
To review next week.
Updated by pjessen about 1 year ago
- Due date changed from 2022-08-23 to 2023-05-17
pjessen wrote:
To review next week.
That went well - I have just today "discovered" much the same issue on provo-mirror, see #128555.