tickets #166475
openForward non-syslog logs to syslog server
0%
Description
We currently do syslog forwarding to our syslog server. This covers the systemd journal via /var/log/messages. Having logs accessible and archived in a central place is generally useful, but only /var/log/messages often does not help. We should additionally forward /var/log/mail and application specific log files.
Currently the structure on our syslog server (monitor.i.o.o) is:
/var/log/opensuse/hosts/<hostname>.log
I think it makes sense to keep this but to extend it further with:
/var/log/opensuse/hosts/<hostname>.log.d/mail.log
/var/log/opensuse/hosts/<hostname>.log.d/<application tag>.log
From my brief research, this will require two changes:
- collection of additional log files on client rsyslogd's using the imfile module (https://www.rsyslog.com/doc/configuration/modules/imfile.html) and either tagging them or making sure the application name is set correctly
- splitting the remote input on the syslog server's rsyslogd by tags in addition to hostnames using an additional template like
template(name="DynFileApp" type="string" string="/var/log/opensuse/hosts/%HOSTNAME%.log.d/%syslogtag%.log")(or%programname%if we have the application names set correctly - https://www.rsyslog.com/doc/configuration/properties.html - adjusting the existing "DynFile => .log" rule to only act on syslog/messages instead of on everything arriving from a host
Our rsyslog server configuration is in https://code.opensuse.org/heroes/salt/blob/production/f/salt/profile/log/files/etc/rsyslog.d/server.conf.
Our rsyslog client configuration is comprised of a few pieces, there is a template in https://code.opensuse.org/heroes/salt/blob/production/f/salt/profile/log/files/etc/rsyslog.d/remote.conf.jinja which is populated through some pillar in https://code.opensuse.org/heroes/salt/blob/production/f/pillar/common/init.sls#_69. There are also some other client side templates which take care of exclusions and mtail which might need consideration to not break when new rules are added.
Updated by cboltz 4 months ago
This idea makes sense, but I wonder if it's a good idea to keep the filenames /var/log/opensuse/hosts/<hostname>.log and to add a <hostname<.log.d directory for "other" logs from this host (which will at least be a small mail log for many hosts if a cronjob fails etc.)
I'd prefer to move the main logs to /var/log/opensuse/hosts/<hostname>/messages.log , and to have the other logs from that host also in that directory.
Updated by crameleon 3 months ago
I evaluated the options for and felt more comfortable replacing rsyslog with syslog-ng as part of implementing this. The configuration format seems cleaner and syntax errors give very useful output. The open source edition cannot do reliable logging with rsyslog clients, but syslog-ng clients can do it when using the OTEL protocol, which would be more modern than traditional syslog anyways. Started hacking on it in this branch: https://gitlab.infra.opensuse.org/infra/salt/-/tree/crameleon/logng, includes some sample filtering and file rules already.