tickets #104712
closedpontifex apache logs - why are there ipv4 client addresses in the ipv6 logs?
Added by pjessen over 2 years ago. Updated 6 months ago.
0%
Description
This is really weird.
The apache logs in /var/log/apache2/202x/x/ipv6.download.opensuse.org-yyyymmdd-access_log contain quite a lot of IPv4 client addresses??
Looking at an old one, ipv6.download.opensuse.org-20211203-access_log.xz, there is only 4725 such lines, but in more recent logs, up to 37% of an ipv6 logfile are ipv4 addresses......
Files
| Screenshot_20220203_180734.jpeg (99.2 KB) Screenshot_20220203_180734.jpeg | wireshark screenshot | pjessen, 2022-02-03 17:11 |
Updated by pjessen over 2 years ago
- Private changed from Yes to No
Take e.g. ipv6.download.opensuse.org-20220103-access_log.xz - of 24'679'402 lines, 9'163'175 are IPv4 client addresses.
Updated by pjessen over 2 years ago
Tangent - we are writing concurrently to the same logfile with two cronolog processes. I presume due to different vhosts for http:// and https:// I am not intimately familiar with cronolog, but is this a supported setup?
Updated by pjessen over 2 years ago
pjessen wrote:
Take e.g. ipv6.download.opensuse.org-20220103-access_log.xz - of 24'679'402 lines, 9'163'175 are IPv4 client addresses.
There are also IPv6 addresses in the IPv4 logfiles, e.g. download.opensuse.org-20220106-access_log.xz:
xzegrep '^[0-9A-Fa-f:]+\ ' download.opensuse.org-20220106-access_log.xz | wc -l
9705
Updated by pjessen over 2 years ago
It's getting better and better - looking at today's log, /var/log/apache2/download.opensuse.org/2022/02/download.opensuse.org-20220201-access_log, I see a number of IPv6 addresses in the 2001:db8 prefix:
2001:db8:116e:8d81:cb7:784a:1aa4:86a8
2001:db8:128a:244a:77ca:40dc:125:12be
2001:db8:13c9:fc9e:1595:b999:5e34:5929
2001:db8:19ec:24c9:5941:2866:125:12be
2001:db8:1b1f:dc03:3a30:e304:3e5d:2e5d
2001:db8:1eea:ac48:2869:7628:678d:808
2001:db8:23a9:98a1:448a:3c9f:4c66:da6
2001:db8:24:e0d3:5665:44e4:7c4a:e855
2001:db8:2906:9193:2f07:ad36:4c66:da6
2001:db8:2e01:2025:287:20bb:7c4a:e855
2001:db8:3528:e4fa:28e4:534d:125:12be
2001:db8:39c3:f84:5903:366:75d9:2c79
2001:db8:3a26:72ef:4bd7:239b:75d9:2c79
2001:db8:3a2c:ec2:4625:1828:3e5d:2e5d
2001:db8:3d42:8e76:1650:8198:1aa4:86a8
2001:db8:3e22:52f8:4c9:2840:75d9:2c79
2001:db8:4303:5748:4c39:2822:75d9:2c79
2001:db8:47c6:5bee:4158:1add:5e34:5929
2001:db8:4d1b:6645:6500:ceff:5e34:5929
2001:db8:4e01:37d0:6aa6:44d4:5e34:5929
2001:db8:5d2b:c5f9:3f8c:607:5e34:5929
2001:db8:5e59:5352:39aa:97ac:125:12be
2001:db8:6188:e226:2e85:4b5a:1aa4:86a8
2001:db8:655d:315d:652b:e334:678d:808
2001:db8:6dc9:417:4fc9:60e3:678d:808
2001:db8:6e66:9646:3f85:b627:125:12be
2001:db8:6eb5:f5b8:1802:2290:125:12be
2001:db8:704c:eb47:629a:f56f:cb3:61c6
2001:db8:7884:a7c0:43e6:96e7:125:12be
2001:db8:7a29:6a55:510e:f519:678d:808
2001:db8:84:8aec:3d23:e1da:678d:808
2001:db8/32 is reserved for documentation purposes, see RFC3849.
Remarkably, these only appear in the ipv4 log, there are none in /var/log/apache2/ipv6.download.opensuse.org/2022/02/ipv6.download.opensuse.org-20220201-access_log
Updated by pjessen over 2 years ago
pjessen wrote:
It's getting better and better - looking at today's log, /var/log/apache2/download.opensuse.org/2022/02/download.opensuse.org-20220201-access_log, I see a number of IPv6 addresses in the 2001:db8 prefix:
This seems to be "normal", going back over the log from December and January, I see 300-500 such requests, daily.
Updated by pjessen over 2 years ago
This morning I ran a tcpdump on any interface, looking for '2001:db8::/32'. Despite the usual entries in the access log, nothing showed up in the dump. I'm not surprised, but it does not help figuring out where those requests are coming from nor how they end up in the wrong log.
Updated by pjessen over 2 years ago
- Related to tickets #105807: 2001:db8:: clients on pontifex ? added
Updated by pjessen over 2 years ago
pjessen wrote:
It's getting better and better - looking at today's log, /var/log/apache2/download.opensuse.org/2022/02/download.opensuse.org-20220201-access_log, I see a number of IPv6 addresses in the 2001:db8 prefix:
This is now being tracked in #105807
Updated by pjessen over 2 years ago
As in #105807, this is being caused by:
<IfModule mod_remoteip.c>
RemoteIPHeader X-Forwarded-For
</IfModule>
We add our own X-Forwarded-For in haproxy on anna, but somehow it is being ignored. From the haproxy configuration manual:
Since this header is always appended at the end of the existing header list, the server must be configured to always use the last occurrence of this header only.
See the server's manual to find how to enable use of this standard header. Note that only the last occurrence of the header must be used, since it is really
possible that the client has already brought one.
Updated by pjessen over 2 years ago
- Assignee set to pjessen
I am very tempted to try the following:
a) change the haproxy config on anna to use 'X-Forwarded-999' (instead of X-Forwarded-For)
b) change apache2 on pontifex to use 'X-Forwarded-999' (with the RemoteIPHeader directive)
Updated by pjessen over 2 years ago
Note - the haproxy config on anna, section 'frontend http-ext-in', does have
http-request del-header ^X-Forwarded-For:.*'
so it seems strange
to have these addresses in X-Forwarded-For. I have a suspicion that the above is wrong though, it should probably be:
http-request del-header X-Forwarded-For
According to the haproxy config manual, the argument to 'del-header' is not a regex, but just the name of the field.
Updated by pjessen over 2 years ago
- Status changed from New to In Progress
Just logging my changes: I have changed haproxy.cfg on anna and elsa to use 'http-request del-header X-Forwarded-For', but sofar it doesn't seem to have made any difference. I don't understand why not.
I picked a random address from my db9_log, 165.225.27.83, and I ran a tcpdump. There is still a 'X-Forwarded-For'.
Updated by pjessen over 2 years ago
A wireshark screenshot clearly showing an http request with X-Forwarded-For.
Updated by pjessen over 2 years ago
Changelog: I have implemented my suggestion from comment#10.
This seems to have had the right kind of effect - looking at about 150.000 requests, I now only see 'X-Forwarded-For' for 195.135.221.145 and 195.135.221.139. This does not seem to solve the issue in this ticket though.
Updated by pjessen over 2 years ago
a) ipv4 addresses in the ipv6 logs - seems to have been resolved, at least there are none from midnight until now (1520UTC)
b) ipv6 addresses in the ipv4 logs - remains a problem. There seems to be less, I'll have to check.
Updated by pjessen about 2 years ago
pjessen wrote:
a) ipv4 addresses in the ipv6 logs - seems to have been resolved, at least there are none from midnight until now (1520UTC)
Confirmed, no more ipv4 addresses in the ipv6 logs, since 4 Feb.
b) ipv6 addresses in the ipv4 logs - remains a problem. There seems to be less, I'll have to check.
12 Feb - 25540300 loglines, of which 20098 with ipv6 addresses.