openSUSE admin

Uh, update - we're not using cronolog, it just looked like it - this is from /etc/nginx/vhosts.d/000-downloadcontent.conf

access_log  /var/log/nginx/downloadcontent/$year/$month/$year-$month-$day-access.log  main;

Something stop&starts nginx at midnight which gets a new log started, but I can't figure out what it is. I don't see any cronjob nor any systemd timer doing that.
Ah, it must be the "/usr/sbin/nginx -s reopen" from logrotate.
Logrotate does the compression but it never cleans out any old logs. I have disabled 'nginx-compress-logs' in /etc/cron.d/nginx-time-based-logfiles-helpers. I have also amended the logrotate config to not use "dateext" with /var/log/nginx/*/*/*/*.log

In addition, every day at 00:30, we also run /usr/sbin/nginx-compress-logs, a small script that does pretty much the same.

The script also updates a symlink.

Occasionally, these errors are reported to admin-auto:

xz: /var/log/nginx/downloadcontent/2022/03/2022-03-09-access.log.xz: File exists
xz: /var/log/nginx/downloadcontent/2022/03/2022-03-12-access.log.xz: File exists

pontifex2 (download.o.o):~ # l /var/log/nginx/downloadcontent/2022/03/
total 158156
drwxr-xr-x 2 nginx nginx     4096 Mar 14 00:30 ./
drwxr-xr-x 5 nginx nginx       33 Mar  1 00:00 ../
-rw-r--r-- 1 nginx nginx      304 Mar  3 01:34 2022-03-02-access.log.xz
-rw-r--r-- 1 nginx nginx  4726920 Mar  6 00:03 2022-03-05-access.log.xz
-rw-r--r-- 1 nginx nginx  4284940 Mar  7 00:17 2022-03-06-access.log.xz
-rw-r--r-- 1 nginx nginx 10060608 Mar  8 00:13 2022-03-07-access.log.xz
-rw-r--r-- 1 nginx nginx  7297828 Mar  9 00:09 2022-03-08-access.log.xz
-rw-r--r-- 1 nginx nginx      339 Mar 10 00:45 2022-03-09-access.log
-rw-r--r-- 1 nginx nginx 12546652 Mar 10 00:09 2022-03-09-access.log.xz
-rw-r--r-- 1 nginx nginx  6964504 Mar 11 00:02 2022-03-10-access.log.xz
-rw-r--r-- 1 nginx nginx  8437036 Mar 12 00:00 2022-03-11-access.log.xz
-rw-r--r-- 1 nginx nginx      354 Mar 13 00:58 2022-03-12-access.log
-rw-r--r-- 1 nginx nginx  5173420 Mar 13 00:20 2022-03-12-access.log.xz
-rw-r--r-- 1 nginx nginx  5750752 Mar 14 00:02 2022-03-13-access.log.xz

2022-03-12-access.log contains one line, with an entry for 2022-03-13.

pjessen wrote:

"Cannot set the file group: Operation not permitted" - apparmor?

I don't see a profile that might be related to logrotation or xz, and audit.log doesn't say anything about it, so AppArmor is a bit ;-) unlikely.

Unfortunately, I don't know what else could cause the permission issue - even if script (and therefore xz) runs as user nginx, it has full permissions on those directories (directory and files owned by nginx:nginx).

(Hmm, maybe add a ls -al in the script before and after compressing the logs? That might give us a hint what's going on.)

.

One thing I noticed is that you disabled nginx-compress-logs in 2019(!), but your change was moved to a *.rpmsave :-/ (the new/active version has MAILTO added, so please don't just move the *.rpmsave back)

pontifex2 (download.o.o):/etc # head -n100 /etc/cron.d/nginx-time-based-logfiles-helpers*
==> /etc/cron.d/nginx-time-based-logfiles-helpers <==
MAILTO=admin-auto@opensuse.org
@monthly   nginx    /usr/sbin/nginx-logdirs
30 0 * * * nginx    /usr/sbin/nginx-compress-logs

==> /etc/cron.d/nginx-time-based-logfiles-helpers.rpmsave <==
@monthly   nginx    /usr/sbin/nginx-logdirs
# 20190610 disabled, logrotate takes care of it. see https://progress.opensuse.org/issues/52823
#30 0 * * * nginx    /usr/sbin/nginx-compress-logs

cboltz wrote:

pjessen wrote:

"Cannot set the file group: Operation not permitted" - apparmor?

I don't see a profile that might be related to logrotation or xz, and audit.log doesn't say anything about it, so AppArmor is a bit ;-) unlikely.

Yeah, it was a wild guess. I did look at the log too and saw nothing.

One thing I noticed is that you disabled nginx-compress-logs in 2019(!), but your change was moved to a *.rpmsave :-/ (the new/active version has MAILTO added, so please don't just move the *.rpmsave back)

Got it. 2019, that was a lifetime ago ... thanks for reminding me.
I have disabled the cronjob again, maintaining the MAILTO. The "issue" of creating the symlink remains, but I think it is really minor. Maybe it could be worked into the logrotate config.