Project

General

Profile

Actions

tickets #50633

closed

/usr/bin/geoip-lite-update - gzip: GeoIP.dat.gz: not in gzip format

Added by pjessen almost 5 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Mirrors
Target version:
-
Start date:
2019-04-23
Due date:
2020-01-20
% Done:

50%

Estimated time:

Description

Judging by my inbox, this has been going since 7 January 2019. It applies to olaf, pontifex and provo-mirror.

It looks like maxmind changed something, so all we are downloading into /var/lib/GeoIP/GeoIP.dat.gz is the HTML 404 we are getting.

# cat GeoIP.dat.gz
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

I'll try an figure out what was changed at maxmind.

Actions #1

Updated by pjessen almost 5 years ago

  • Private changed from Yes to No
Actions #2

Updated by pjessen almost 5 years ago

https://support.maxmind.com/geolite-legacy-discontinuation-notice/

On January 2, 2019, we discontinued our GeoLite Legacy databases. This discontinuation was first announced on January 2, 2018.

In /var/lib/GeoIP I see three Maxmind databases: GeoIP.dat.gz, GeoIPv6.dat.gz, GeoLiteCity.dat.gz. Looking at maxmind geolite2 information
https://dev.maxmind.com/geoip/geoip2/geolite2/, there are now

Geolite2 City - https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
GeoLite2 COuntry - https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz
Geolite2 ASN -  https://geolite.maxmind.com/download/geoip/database/GeoLite2-ASN.tar.gz
Actions #3

Updated by pjessen almost 5 years ago

These databases are used by mirrorbrain by way of apache, /etc/apache2/vhosts.d/_download.conf.macro :

<VirtualHost 195.135.221.134:80>
Use mirrorbrain "download.opensuse.org" "/var/lib/GeoIP/GeoLiteCity.dat.updated" "mirrordb2-nossl-v4.infra.opensuse.org" "15"
</VirtualHost>
<VirtualHost [2620:113:80c0:8::13]:80>
Use mirrorbrain "ipv6.download.opensuse.org" "/var/lib/GeoIP/GeoIPv6.dat.updated" "mirrordb2-nossl-v6.infra.opensuse.org" "15"
</VirtualHost>
<VirtualHost [2001:67c:2178:8::13]:80>
Use mirrorbrain "ipv6.download.opensuse.org" "/var/lib/GeoIP/GeoIPv6.dat.updated" "mirrordb2-nossl-v6-2.infra.opensuse.org" "15"
</VirtualHost>

<VirtualHost 195.135.221.134:443>
Use mirrorbrain "download.opensuse.org" "/var/lib/GeoIP/GeoLiteCity.dat.updated" "mirrordb2-ssl-v4.infra.opensuse.org" "16"
Use ssl_settings
</VirtualHost>
<VirtualHost [2620:113:80c0:8::13]:443>
Use mirrorbrain "ipv6.download.opensuse.org" "/var/lib/GeoIP/GeoIPv6.dat.updated" "mirrordb2-ssl-v6.infra.opensuse.org" "16"
Use ssl_settings
</VirtualHost>
<VirtualHost [2001:67c:2178:8::13]:443>
Use mirrorbrain "ipv6.download.opensuse.org" "/var/lib/GeoIP/GeoIPv6.dat.updated" "mirrordb2-ssl-v6-2.infra.opensuse.org" "16"
Use ssl_settings
</VirtualHost>

Actions #4

Updated by pjessen almost 5 years ago

The new tar.gz files contain the data in mmdb format, the format of which may be found here: https://maxmind.github.io/MaxMind-DB/

Actions #5

Updated by pjessen almost 5 years ago

pjessen wrote:

The new tar.gz files contain the data in mmdb format, the format of which may be found here: https://maxmind.github.io/MaxMind-DB/

I see no easy way of dumping one of these into the format we need, but fortunately MaxMind also provides the data in CSV format:

Geolite2 City - https://geolite.maxmind.com/download/geoip/database/GeoLite2-City-CSV.zip
GeoLite2 COuntry - https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country-CSV.zip
Geolite2 ASN -  https://geolite.maxmind.com/download/geoip/database/GeoLite2-CSV.zip

Contents of GeoLite2-Country-CSV.gz:

unzip -l GeoLite2-Country-CSV.zip
Archive:  GeoLite2-Country-CSV.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
    15004  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-ja.csv
    11422  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-pt-BR.csv
      433  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/LICENSE.txt
     9948  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-en.csv
 11953015  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Blocks-IPv4.csv
     9996  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-es.csv
    11735  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-zh-CN.csv
    10234  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-fr.csv
  3644577  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Blocks-IPv6.csv
       55  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/COPYRIGHT.txt
     9836  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-de.csv
    14955  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/GeoLite2-Country-Locations-ru.csv
      116  2019-06-11 22:23   GeoLite2-Country-CSV_20190611/README.txt
---------                     -------
 15691326                     13 files
Actions #6

Updated by pjessen almost 5 years ago

The Maxmind databases are used by apache module "mod_geoip", alternatively by "mod_mirrorbrain" (only used if mod_geoip is not available).
mod_geoip is supplied my maxmind, but has been superceded by "mod_maxminddb": https://github.com/maxmind/mod_maxminddb. I'll check if anyone has already built this for Leap 42.3, otherwise I'll build from source.

Actions #7

Updated by pjessen about 4 years ago

  • Due date set to 2020-01-20
  • Status changed from New to In Progress
  • % Done changed from 0 to 50

Looks like I forgot to update this - Pontifex was since updated to 15.1, and mod_maxminddb is installed.
The same problem applies on olaf, but I am not sure how the geoip databases are used here.

Actions #8

Updated by pjessen about 4 years ago

  • Status changed from In Progress to Resolved

I have updated the procedure for refreshing the Maxmind databases on olaf, but as I don't know how they are used, I have no way of verifying that it works :-(

Actions

Also available in: Atom PDF