tickets #50633
closed/usr/bin/geoip-lite-update - gzip: GeoIP.dat.gz: not in gzip format
Added by pjessen about 5 years ago. Updated over 4 years ago.
50%
Description
Judging by my inbox, this has been going since 7 January 2019. It applies to olaf, pontifex and provo-mirror.
It looks like maxmind changed something, so all we are downloading into /var/lib/GeoIP/GeoIP.dat.gz is the HTML 404 we are getting.
# cat GeoIP.dat.gz
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
I'll try an figure out what was changed at maxmind.
Updated by pjessen about 5 years ago
https://support.maxmind.com/geolite-legacy-discontinuation-notice/
On January 2, 2019, we discontinued our GeoLite Legacy databases. This discontinuation was first announced on January 2, 2018.
In /var/lib/GeoIP I see three Maxmind databases: GeoIP.dat.gz, GeoIPv6.dat.gz, GeoLiteCity.dat.gz. Looking at maxmind geolite2 information
https://dev.maxmind.com/geoip/geoip2/geolite2/, there are now
Geolite2 City - https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
GeoLite2 COuntry - https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz
Geolite2 ASN - https://geolite.maxmind.com/download/geoip/database/GeoLite2-ASN.tar.gz
Updated by pjessen about 5 years ago
These databases are used by mirrorbrain by way of apache, /etc/apache2/vhosts.d/_download.conf.macro :
<VirtualHost 195.135.221.134:80>
Use mirrorbrain "download.opensuse.org" "/var/lib/GeoIP/GeoLiteCity.dat.updated" "mirrordb2-nossl-v4.infra.opensuse.org" "15"
</VirtualHost>
<VirtualHost [2620:113:80c0:8::13]:80>
Use mirrorbrain "ipv6.download.opensuse.org" "/var/lib/GeoIP/GeoIPv6.dat.updated" "mirrordb2-nossl-v6.infra.opensuse.org" "15"
</VirtualHost>
<VirtualHost [2001:67c:2178:8::13]:80>
Use mirrorbrain "ipv6.download.opensuse.org" "/var/lib/GeoIP/GeoIPv6.dat.updated" "mirrordb2-nossl-v6-2.infra.opensuse.org" "15"
</VirtualHost>
<VirtualHost 195.135.221.134:443>
Use mirrorbrain "download.opensuse.org" "/var/lib/GeoIP/GeoLiteCity.dat.updated" "mirrordb2-ssl-v4.infra.opensuse.org" "16"
Use ssl_settings
</VirtualHost>
<VirtualHost [2620:113:80c0:8::13]:443>
Use mirrorbrain "ipv6.download.opensuse.org" "/var/lib/GeoIP/GeoIPv6.dat.updated" "mirrordb2-ssl-v6.infra.opensuse.org" "16"
Use ssl_settings
</VirtualHost>
<VirtualHost [2001:67c:2178:8::13]:443>
Use mirrorbrain "ipv6.download.opensuse.org" "/var/lib/GeoIP/GeoIPv6.dat.updated" "mirrordb2-ssl-v6-2.infra.opensuse.org" "16"
Use ssl_settings
</VirtualHost>
Updated by pjessen almost 5 years ago
The new tar.gz files contain the data in mmdb format, the format of which may be found here: https://maxmind.github.io/MaxMind-DB/
Updated by pjessen almost 5 years ago
pjessen wrote:
The new tar.gz files contain the data in mmdb format, the format of which may be found here: https://maxmind.github.io/MaxMind-DB/
I see no easy way of dumping one of these into the format we need, but fortunately MaxMind also provides the data in CSV format:
Geolite2 City - https://geolite.maxmind.com/download/geoip/database/GeoLite2-City-CSV.zip
GeoLite2 COuntry - https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country-CSV.zip
Geolite2 ASN - https://geolite.maxmind.com/download/geoip/database/GeoLite2-CSV.zip
Contents of GeoLite2-Country-CSV.gz:
unzip -l GeoLite2-Country-CSV.zip
Archive: GeoLite2-Country-CSV.zip
Length Date Time Name
--------- ---------- ----- ----
15004 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-ja.csv
11422 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-pt-BR.csv
433 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/LICENSE.txt
9948 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-en.csv
11953015 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Blocks-IPv4.csv
9996 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-es.csv
11735 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-zh-CN.csv
10234 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-fr.csv
3644577 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Blocks-IPv6.csv
55 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/COPYRIGHT.txt
9836 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-de.csv
14955 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-ru.csv
116 2019-06-11 22:23 GeoLite2-Country-CSV_20190611/README.txt
--------- -------
15691326 13 files
Updated by pjessen almost 5 years ago
The Maxmind databases are used by apache module "mod_geoip", alternatively by "mod_mirrorbrain" (only used if mod_geoip is not available).
mod_geoip is supplied my maxmind, but has been superceded by "mod_maxminddb": https://github.com/maxmind/mod_maxminddb. I'll check if anyone has already built this for Leap 42.3, otherwise I'll build from source.
Updated by pjessen over 4 years ago
- Due date set to 2020-01-20
- Status changed from New to In Progress
- % Done changed from 0 to 50
Looks like I forgot to update this - Pontifex was since updated to 15.1, and mod_maxminddb is installed.
The same problem applies on olaf, but I am not sure how the geoip databases are used here.
Updated by pjessen over 4 years ago
- Status changed from In Progress to Resolved
I have updated the procedure for refreshing the Maxmind databases on olaf, but as I don't know how they are used, I have no way of verifying that it works :-(