Project

General

Profile

tickets #62678

pontifex - mirrorbrain - routeviews - ip4r does not support ipv6 addresses.

Added by pjessen about 3 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Mirrors
Target version:
-
Start date:
2020-01-26
Due date:
% Done:

100%

Estimated time:

Description

Every day we download two snapshots from mirrorbrain.org with ipv4 and ipv6 route information. These are loaded into table "pfx2asn". Unfortunately, the 'pfx^ column is of type 'ip4r' which only accepts ipv4 ranges.

When the daily job reports:

Downloading http://mirrorbrain.org/routeviews/oix-full-snapshot-latest.dat.bz2
nothing imported, no change comitted to the database

That is really a problem - it means our pfx2asn table was NOT updated, but it should ALWAYS be updated.

Traceback (most recent call last):
File "/tmp/asn_imp2", line 95, in
import_raw()
File "/tmp/asn_imp2", line 74, in import_raw
cursor.execute("INSERT INTO %s VALUES ( %%s, %%s )" % tablename, [pfx, asn])
psycopg2.DataError: invalid IP4R value: "2001::/32"
LINE 1: INSERT INTO pfx2asn VALUES ( '2001::/32', 1101 )

Therefore, the import fails and ALL the changes are rolled back.

I am wondering if maybe asn_get_routeview and asn_import were updated (in apache2-mod_asn-tools) without our database being updated?

I hesitate suggesting changing the column type to 'iprange' which accepts both ipv4 and ipv6 ranges, I have no idea what it might affect.

History

#1 Updated by pjessen about 3 years ago

  • Assignee set to pjessen
  • Private changed from Yes to No

To start with, I'll just remove the file from the download. See /usr/bin/asn_get_routeviews.

#2 Updated by pjessen about 3 years ago

  • Status changed from New to In Progress

pjessen wrote:

To start with, I'll just remove the file from the download. See /usr/bin/asn_get_routeviews.

Correction - looking at the setup sql in apache2-mod_asn :

CREATE TABLE "pfx2asn" (
"pfx" iprange NOT NULL PRIMARY KEY,
"asn" integer NOT NULL
);

the schema was updated to use 'iprange', but when we updated pontifex, this was not spotted. I suggest this problem is best fixed by just ALTERing that column type from 'ip4r' to 'iprange'.

#3 Updated by pjessen about 3 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

I have updated the column type to 'iprange'. It was automatically propagated from mirrordb1 to mirrordb2. I did a reload of the pfx2asn table, worked fine.

#4 Updated by pjessen about 3 years ago

  • Status changed from Resolved to New

Reopening. I tried to add some new mirrors this morning, "mb new" complained about ip4r_size being used in an ORDER BY clause.
This seems to be in /usr/lib64/python2.7/site-packages/mb/asn.py, from package python-mb.

I have updated asn.py, line 53, to use iprange instead of ip4r and iprange_size instead of ip4r_size.

#5 Updated by pjessen almost 3 years ago

  • Status changed from New to In Progress

I don't know if this is related, but -

looking at the weblogs for 20200105, and only requests for distribution/leap/15.1/repo (to limit the amount of data), I see 23693 unique address of which 20128 did not have any geolocation. That can't be right. Looking at the aame for 20200204, I also see addresses sometimes correctly geolocated, sometimes not. E.g. 102.141.173.49 - this is an address in South Africa, so should read 'AF:ZA'. In the logs from that date, 14 it is correct, 54 times it is not.

#6 Updated by pjessen almost 3 years ago

AFAICT, this is not related to my fixing the routeviews problem. The country and continent are looked up my the maxmind module. I think I will open a separate issue.

#7 Updated by darix almost 3 years ago

you can drop mod_asn and related tools from pontifex. we can leave postgresqlXY-ip4r for now but it is only used in the mod_asn table IIRC

the whole ASN and Geo data lookup is now handled in via mod_maxminddb.

<IfModule mod_maxminddb.c>
        MaxMindDBEnable On

        # ###
        # specify database locations
        #
        MaxMindDBFile CITY_DB    /etc/apache2/dbs/GeoLite2-City.mmdb
        MaxMindDBFile ASN_DB     /etc/apache2/dbs/GeoLite2-ASN.mmdb

        # ###
        # assign the lookup result to an environment variable
        #
        #                   variable              DB name/selector
        #
        MaxMindDBEnv        GEOIP_COUNTRY_CODE    CITY_DB/country/iso_code
        MaxMindDBEnv        GEOIP_COUNTRY_NAME    CITY_DB/country/names/en
        MaxMindDBEnv        GEOIP_CONTINENT_CODE  CITY_DB/continent/code
        MaxMindDBEnv        GEOIP_LATITUDE        CITY_DB/location/latitude
        MaxMindDBEnv        GEOIP_LONGITUDE       CITY_DB/location/longitude
        MaxMindDBEnv        ASN                   ASN_DB/autonomous_system_number


        #                   DB name               variable
        #
        MaxMindDBNetworkEnv ASN_DB                PFX
        </IfModule>

With the recent upgrade of libmaxminddb and mod_maxminddb even the PFX match is working again.

#8 Updated by pjessen almost 3 years ago

darix wrote:

you can drop mod_asn and related tools from pontifex. we can leave postgresqlXY-ip4r for now but it is only used in the mod_asn table IIRC

In table pfx2asn, I changed the column to 'iprange', back in January. So I don't even need to run asn_get_routeview etc?

the whole ASN and Geo data lookup is now handled in via mod_maxminddb.

Ah, that's cool.

MaxMindDBEnable On

    # ###
    # specify database locations
    #
    MaxMindDBFile CITY_DB    /etc/apache2/dbs/GeoLite2-City.mmdb
    MaxMindDBFile ASN_DB     /etc/apache2/dbs/GeoLite2-ASN.mmdb

I guess you've found ours in /var/lib/GeoIP.

#9 Updated by lrupp about 1 year ago

  • Status changed from In Progress to Closed

Hi there - and a Happy and Healthy 2022!

I'm currently closing old tickets which did not see much change.
If the main concern still exists and should be handled, please re-open by just replying to this Email.

Thanks in advance,
Lars

Also available in: Atom PDF