openSUSE admin

Same thing for #90335, mirrors.163.com

See also #92611 - a UK mirror, but when I ran an "mb update --all gethosted", the region code was cleared and country set to 'tw'. Is this an issue with the maxmind geoip info, somehow?

Just some notes for myself -

• the maxmind data in /var/lib/GeoIP is being updated regularly, seems fine.
• we don't seem to be doing "asn_get_routeviews | asn_import -b mirrordb1" anymore, it was disabled last year. Lars wrote "not needed any longer, as we are using mod_maxminddb now".
  

pjessen wrote:

• we don't seem to be doing "asn_get_routeviews | asn_import -b mirrordb1" anymore, it was disabled last year. Lars wrote "not needed any longer, as we are using mod_maxminddb now".

I think that is the problem - 'mb' depends on that data to assign something useful to region and country when creating or updating a mirror.
I tried running "asn_get_routeviews | asn_import -b mirrordb1" manually, but it fails:

asn_get_routeviews | asn_import -b mirrordb2
Using existing file "oix-full-snapshot-latest.dat.bz2", because it is less than 8h old.
Remove it to have it downloaded again.
Using existing file "ipv6-rib-snapshot-latest.txt.bz2", because it is less than 8h old.
Remove it to have it downloaded again.
Traceback (most recent call last):
  File "/usr/bin/asn_import", line 27, in <module>
    connection = mb.conn.Conn(config.dbconfig)
TypeError: __init__() takes at least 3 arguments (2 given)
[Errno 32] Broken pipe

Somehow the mb.conn.Conn() interface is no longer compatible? it now needs a 'version' argument too, which asn_import does not provide.

When someone calls "mb update --something gethosted", it uses the mb/geoip.py module, which in turn uses 'geoiplookup' and 'geoiplookup_continent' - these appear to have been updated to use the maxmind geoip data, good.

However, 'geoiplookup_continent' returns nothing ...

geoiplookup_continent -f mirrordb1 185.215.215.201
geoiplookup_continent -f mirrordb1 185.85.248.1
geoiplookup_continent -f mirrordb2 185.85.248.1
geoiplookup_continent -f mirrordb2 195.135.221.130

No output.

/usr/bin/geoiplookup_continent has a couple of typos, line 141:

if script_name == 'geoiplookup_contintent':
  print(mb_host.country_code())

It should most probably be:

if script_name == 'geoiplookup_continent':
  print(mb_host.region_code())

pjessen wrote:

• we don't seem to be doing "asn_get_routeviews | asn_import -b mirrordb1" anymore, it was disabled last year. Lars wrote "not needed any longer, as we are using mod_maxminddb now".

I think that is the problem - 'mb' depends on that data to assign something useful to region and country when creating or updating a mirror.

Hmm, no. The geoiplookup utilities were updated to work with maxmind.

pjessen wrote:

When someone calls "mb update --something gethosted", it uses the mb/geoip.py module, which in turn uses 'geoiplookup' and 'geoiplookup_continent' - these appear to have been updated to use the maxmind geoip data, good.

The mb/geoip.py module uses the following files (hardcoded) :

/var/lib/GeoIP/GeoLiteCity.dat.updated
/var/lib/GeoIP/GeoLiteCity.dat
/var/lib/GeoIP/GeoIP.dat.updated
/var/lib/GeoIP/GeoIP.dat
/usr/share/GeoIP/GeoLiteCity.dat.updated
/usr/share/GeoIP/GeoLiteCity.dat
/usr/share/GeoIP/GeoIP.dat.updated
/usr/share/GeoIP/GeoIP.dat
/var/lib/GeoIP/GeoIPv6.dat.updated
/var/lib/GeoIP/GeoIPv6.dat
/usr/share/GeoIP/GeoIPv6.dat.updated
/usr/share/GeoIP/GeoIPv6.dat

The only ones of those that exist are:

-rw-r--r-- 1 root root  2099199 Dec 22  2019 /var/lib/GeoIP/GeoIP.dat
-rw-r--r-- 1 root root  1242574 Dec 31  2018 /var/lib/GeoIP/GeoIP.dat.updated
-rw-r--r-- 1 root root 20539238 Dec 31  2018 /var/lib/GeoIP/GeoLiteCity.dat.updated
-rw-r--r-- 1 root root  2456195 Dec 31  2018 /var/lib/GeoIP/GeoIPv6.dat.updated

and as can be seen, they are ancient.

I know I wrote "The geoiplookup utilities were updated to work with maxmind", but it looks like only 'geoiplookup_continent' was updated to at least use the maxmind database files.

https://github.com/openSUSE/mirrorbrain/issues/62

Fixed manually.

Trying to update the ASN:

mb update -a gethosted
mirror.gethosted.online: STRANGE! There's no ASN containing this hosts IP address (194.156.79.82)...

Unfortunately, /usr/bin/mb uses the pfx2asn table which we are no longer updating ...

Trying to update the region:

mb update --region gethosted
mirror.gethosted.online: updating region (eu -> gb)

Trying to update the countrycode:

mb update --country gethosted
mirror.gethosted.online: updating country (gb -> ip address not found)
Traceback (most recent call last):
  File "/usr/bin/mb", line 1729, in <module>
    r = mirrordoctor.main()
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 261, in main
    return self.cmd(args)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 284, in cmd
    retval = self.onecmd(argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 422, in onecmd
    return self._dispatch_cmd(handler, argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 1123, in _dispatch_cmd
    return handler(argv[0], opts, *args)
  File "/usr/bin/mb", line 485, in do_update
    mirror.country = country
  File "<string>", line 1, in <lambda>
  File "/usr/lib/python2.7/site-packages/sqlobject/main.py", line 1107, in _SO_setValue
    dbValue)])
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 614, in _SO_update
    self.sqlrepr(so.id)))
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 435, in query
    return self._runWithConnection(self._query, s)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 342, in _runWithConnection
    val = meth(conn, *args)
  File "/usr/lib/python2.7/site-packages/sqlobject/dbconnection.py", line 431, in _query
    self._executeRetry(conn, c, s)
  File "/usr/lib/python2.7/site-packages/sqlobject/postgres/pgconnection.py", line 259, in _executeRetry
    raise dberrors.DataError(ErrorMessage(e))
sqlobject.dberrors.DataError: value too long for type character varying(2)

pjessen wrote:

Trying to update the ASN:

mb update -a gethosted
mirror.gethosted.online: STRANGE! There's no ASN containing this hosts IP address (194.156.79.82)...

Unfortunately, /usr/bin/mb uses the pfx2asn table which we are no longer updating ...

I guess the pfx2asn table could be updated from the maxmind data. (/var/lib/GeoIP/GeoLite2-ASN.mmdb)

mb_opensuse2=# \d pfx2asn
              Table "public.pfx2asn"
 Column |  Type   | Collation | Nullable | Default 
--------+---------+-----------+----------+---------
 pfx    | iprange |           | not null | 
 asn    | integer |           | not null | 
Indexes:
    "pfx2asn_pfx_key" gist (pfx)

pjessen wrote:

Unfortunately, /usr/bin/mb uses the pfx2asn table which we are no longer updating ...

I guess the pfx2asn table could be updated from the maxmind data. (/var/lib/GeoIP/GeoLite2-ASN.mmdb)

Nope, that database does not have the prefix information. I'm going to re-enable the cronjob that imports the pfx2asn data. See also #61789

With table 'pfx2asn' populated with up-to-date data, it is now possible to assign the correct ASN to a mirror:

# mb update -a gethosted
mirror.gethosted.online: updating autonomous system number (2856 -> 212728)
# mb update -a mirror.sjtu.edu.cn
mirror.sjtu.edu.cn: updating autonomous system number (24364 -> 4538)

The issue with the region has not been fixed yet, though.

pjessen wrote:

Somehow the mb.conn.Conn() interface is no longer compatible? it now needs a 'version' argument too, which asn_import does not provide.

See https://github.com/openSUSE/mirrorbrain/issues/69