openSUSE admin

Tangent - we are writing concurrently to the same logfile with two cronolog processes. I presume due to different vhosts for http:// and https:// I am not intimately familiar with cronolog, but is this a supported setup?

pjessen wrote:

Take e.g. ipv6.download.opensuse.org-20220103-access_log.xz - of 24'679'402 lines, 9'163'175 are IPv4 client addresses.

There are also IPv6 addresses in the IPv4 logfiles, e.g. download.opensuse.org-20220106-access_log.xz:

xzegrep '^[0-9A-Fa-f:]+\ ' download.opensuse.org-20220106-access_log.xz | wc -l
9705

It's getting better and better - looking at today's log, /var/log/apache2/download.opensuse.org/2022/02/download.opensuse.org-20220201-access_log, I see a number of IPv6 addresses in the 2001:db8 prefix:

2001:db8:116e:8d81:cb7:784a:1aa4:86a8
2001:db8:128a:244a:77ca:40dc:125:12be
2001:db8:13c9:fc9e:1595:b999:5e34:5929
2001:db8:19ec:24c9:5941:2866:125:12be
2001:db8:1b1f:dc03:3a30:e304:3e5d:2e5d
2001:db8:1eea:ac48:2869:7628:678d:808
2001:db8:23a9:98a1:448a:3c9f:4c66:da6
2001:db8:24:e0d3:5665:44e4:7c4a:e855
2001:db8:2906:9193:2f07:ad36:4c66:da6
2001:db8:2e01:2025:287:20bb:7c4a:e855
2001:db8:3528:e4fa:28e4:534d:125:12be
2001:db8:39c3:f84:5903:366:75d9:2c79
2001:db8:3a26:72ef:4bd7:239b:75d9:2c79
2001:db8:3a2c:ec2:4625:1828:3e5d:2e5d
2001:db8:3d42:8e76:1650:8198:1aa4:86a8
2001:db8:3e22:52f8:4c9:2840:75d9:2c79
2001:db8:4303:5748:4c39:2822:75d9:2c79
2001:db8:47c6:5bee:4158:1add:5e34:5929
2001:db8:4d1b:6645:6500:ceff:5e34:5929
2001:db8:4e01:37d0:6aa6:44d4:5e34:5929
2001:db8:5d2b:c5f9:3f8c:607:5e34:5929
2001:db8:5e59:5352:39aa:97ac:125:12be
2001:db8:6188:e226:2e85:4b5a:1aa4:86a8
2001:db8:655d:315d:652b:e334:678d:808
2001:db8:6dc9:417:4fc9:60e3:678d:808
2001:db8:6e66:9646:3f85:b627:125:12be
2001:db8:6eb5:f5b8:1802:2290:125:12be
2001:db8:704c:eb47:629a:f56f:cb3:61c6
2001:db8:7884:a7c0:43e6:96e7:125:12be
2001:db8:7a29:6a55:510e:f519:678d:808
2001:db8:84:8aec:3d23:e1da:678d:808

2001:db8/32 is reserved for documentation purposes, see RFC3849.
Remarkably, these only appear in the ipv4 log, there are none in /var/log/apache2/ipv6.download.opensuse.org/2022/02/ipv6.download.opensuse.org-20220201-access_log

pjessen wrote:

It's getting better and better - looking at today's log, /var/log/apache2/download.opensuse.org/2022/02/download.opensuse.org-20220201-access_log, I see a number of IPv6 addresses in the 2001:db8 prefix:

This seems to be "normal", going back over the log from December and January, I see 300-500 such requests, daily.

This morning I ran a tcpdump on any interface, looking for '2001:db8::/32'. Despite the usual entries in the access log, nothing showed up in the dump. I'm not surprised, but it does not help figuring out where those requests are coming from nor how they end up in the wrong log.

pjessen wrote:

It's getting better and better - looking at today's log, /var/log/apache2/download.opensuse.org/2022/02/download.opensuse.org-20220201-access_log, I see a number of IPv6 addresses in the 2001:db8 prefix:

This is now being tracked in #105807

As in #105807, this is being caused by:

<IfModule mod_remoteip.c>
        RemoteIPHeader X-Forwarded-For
</IfModule>

We add our own X-Forwarded-For in haproxy on anna, but somehow it is being ignored. From the haproxy configuration manual:

Since this header is always appended at the end of the existing header list, the server must be configured to always use the last occurrence of this header only.
See the server's manual to find how to enable use of this standard header. Note that only the last occurrence of the header must be used, since it is really
possible that the client has already brought one.

Note - the haproxy config on anna, section 'frontend http-ext-in', does have

http-request del-header ^X-Forwarded-For:.*'

so it seems strange
to have these addresses in X-Forwarded-For. I have a suspicion that the above is wrong though, it should probably be:

http-request del-header X-Forwarded-For

According to the haproxy config manual, the argument to 'del-header' is not a regex, but just the name of the field.

Changelog: I have implemented my suggestion from comment#10.

This seems to have had the right kind of effect - looking at about 150.000 requests, I now only see 'X-Forwarded-For' for 195.135.221.145 and 195.135.221.139. This does not seem to solve the issue in this ticket though.

a) ipv4 addresses in the ipv6 logs - seems to have been resolved, at least there are none from midnight until now (1520UTC)
b) ipv6 addresses in the ipv4 logs - remains a problem. There seems to be less, I'll have to check.

pjessen wrote:

a) ipv4 addresses in the ipv6 logs - seems to have been resolved, at least there are none from midnight until now (1520UTC)

Confirmed, no more ipv4 addresses in the ipv6 logs, since 4 Feb.

b) ipv6 addresses in the ipv4 logs - remains a problem. There seems to be less, I'll have to check.

12 Feb - 25540300 loglines, of which 20098 with ipv6 addresses.