tickets #62678
closedpontifex - mirrorbrain - routeviews - ip4r does not support ipv6 addresses.
100%
Description
Every day we download two snapshots from mirrorbrain.org with ipv4 and ipv6 route information. These are loaded into table "pfx2asn". Unfortunately, the 'pfx^ column is of type 'ip4r' which only accepts ipv4 ranges.
When the daily job reports:
Downloading http://mirrorbrain.org/routeviews/oix-full-snapshot-latest.dat.bz2
nothing imported, no change comitted to the database
That is really a problem - it means our pfx2asn table was NOT updated, but it should ALWAYS be updated.
Traceback (most recent call last):
File "/tmp/asn_imp2", line 95, in
import_raw()
File "/tmp/asn_imp2", line 74, in import_raw
cursor.execute("INSERT INTO %s VALUES ( %%s, %%s )" % tablename, [pfx, asn])
psycopg2.DataError: invalid IP4R value: "2001::/32"
LINE 1: INSERT INTO pfx2asn VALUES ( '2001::/32', 1101 )
Therefore, the import fails and ALL the changes are rolled back.
I am wondering if maybe asn_get_routeview and asn_import were updated (in apache2-mod_asn-tools) without our database being updated?
I hesitate suggesting changing the column type to 'iprange' which accepts both ipv4 and ipv6 ranges, I have no idea what it might affect.
Updated by pjessen about 4 years ago
- Assignee set to pjessen
- Private changed from Yes to No
To start with, I'll just remove the file from the download. See /usr/bin/asn_get_routeviews.
Updated by pjessen about 4 years ago
- Status changed from New to In Progress
pjessen wrote:
To start with, I'll just remove the file from the download. See /usr/bin/asn_get_routeviews.
Correction - looking at the setup sql in apache2-mod_asn :
CREATE TABLE "pfx2asn" (
"pfx" iprange NOT NULL PRIMARY KEY,
"asn" integer NOT NULL
);
the schema was updated to use 'iprange', but when we updated pontifex, this was not spotted. I suggest this problem is best fixed by just ALTERing that column type from 'ip4r' to 'iprange'.
Updated by pjessen about 4 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
I have updated the column type to 'iprange'. It was automatically propagated from mirrordb1 to mirrordb2. I did a reload of the pfx2asn table, worked fine.
Updated by pjessen about 4 years ago
- Status changed from Resolved to New
Reopening. I tried to add some new mirrors this morning, "mb new" complained about ip4r_size being used in an ORDER BY clause.
This seems to be in /usr/lib64/python2.7/site-packages/mb/asn.py, from package python-mb.
I have updated asn.py, line 53, to use iprange instead of ip4r and iprange_size instead of ip4r_size.
Updated by pjessen about 4 years ago
- Status changed from New to In Progress
I don't know if this is related, but -
looking at the weblogs for 20200105, and only requests for distribution/leap/15.1/repo (to limit the amount of data), I see 23693 unique address of which 20128 did not have any geolocation. That can't be right. Looking at the aame for 20200204, I also see addresses sometimes correctly geolocated, sometimes not. E.g. 102.141.173.49 - this is an address in South Africa, so should read 'AF:ZA'. In the logs from that date, 14 it is correct, 54 times it is not.
Updated by pjessen about 4 years ago
AFAICT, this is not related to my fixing the routeviews problem. The country and continent are looked up my the maxmind module. I think I will open a separate issue.
Updated by darix almost 4 years ago
you can drop mod_asn and related tools from pontifex. we can leave postgresqlXY-ip4r for now but it is only used in the mod_asn table IIRC
the whole ASN and Geo data lookup is now handled in via mod_maxminddb.
<IfModule mod_maxminddb.c>
MaxMindDBEnable On
# ###
# specify database locations
#
MaxMindDBFile CITY_DB /etc/apache2/dbs/GeoLite2-City.mmdb
MaxMindDBFile ASN_DB /etc/apache2/dbs/GeoLite2-ASN.mmdb
# ###
# assign the lookup result to an environment variable
#
# variable DB name/selector
#
MaxMindDBEnv GEOIP_COUNTRY_CODE CITY_DB/country/iso_code
MaxMindDBEnv GEOIP_COUNTRY_NAME CITY_DB/country/names/en
MaxMindDBEnv GEOIP_CONTINENT_CODE CITY_DB/continent/code
MaxMindDBEnv GEOIP_LATITUDE CITY_DB/location/latitude
MaxMindDBEnv GEOIP_LONGITUDE CITY_DB/location/longitude
MaxMindDBEnv ASN ASN_DB/autonomous_system_number
# DB name variable
#
MaxMindDBNetworkEnv ASN_DB PFX
</IfModule>
With the recent upgrade of libmaxminddb and mod_maxminddb even the PFX match is working again.
Updated by pjessen almost 4 years ago
darix wrote:
you can drop mod_asn and related tools from pontifex. we can leave postgresqlXY-ip4r for now but it is only used in the mod_asn table IIRC
In table pfx2asn, I changed the column to 'iprange', back in January. So I don't even need to run asn_get_routeview etc?
the whole ASN and Geo data lookup is now handled in via mod_maxminddb.
Ah, that's cool.
MaxMindDBEnable On
# ### # specify database locations # MaxMindDBFile CITY_DB /etc/apache2/dbs/GeoLite2-City.mmdb MaxMindDBFile ASN_DB /etc/apache2/dbs/GeoLite2-ASN.mmdb
I guess you've found ours in /var/lib/GeoIP.
Updated by lrupp over 2 years ago
- Status changed from In Progress to Closed
Hi there - and a Happy and Healthy 2022!
I'm currently closing old tickets which did not see much change.
If the main concern still exists and should be handled, please re-open by just replying to this Email.
Thanks in advance,
Lars