openSUSE admin

I ran a makehash with -v and found the file causing the problem:

/srv/ftp-stage/pub/opensuse/repositories/home:/legoktm/Debian_9.0/mediawiki_1.31.1.orig.tar.gz

Prior to that one, 93 other files had their hashes updated. The file exists in table 'filearr':

    id     |                                path                                |            mirrors            |                dirname                 |           filename           
-----------+--------------------------------------------------------------------+-------------------------------+----------------------------------------+------------------------------
 230646335 | repositories/home:/legoktm/Debian_9.0/mediawiki_1.31.1.orig.tar.gz | {495,582,566,169,508,583,177} | repositories/home:/legoktm/Debian_9.0/ | mediawiki_1.31.1.orig.tar.gz

Yesterday, I deleted the entry (id 230646335) from the 'hash' table and started a manual makehashes run. It ran for about five hours, until:

Hashing '/srv/ftp-stage/pub/opensuse/repositories/home:/legoktm/Debian_9.0/mediawiki_1.31.1.orig.tar.gz'... 
Traceback (most recent call last):
done.
  File "/usr/bin/mb", line 1729, in <module>
    r = mirrordoctor.main()
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 261, in main
    return self.cmd(args)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 284, in cmd
    retval = self.onecmd(argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 422, in onecmd
    return self._dispatch_cmd(handler, argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 1123, in _dispatch_cmd
    return handler(argv[0], opts, *args)
  File "/usr/bin/mb", line 1148, in do_makehashes
    force=opts.force)
  File "/usr/lib64/python2.7/site-packages/mb/hashes.py", line 175, in check_db
    binascii.hexlify(''.join(self.hb.zsums))]
ValueError: A string literal cannot contain NUL (0x00) characters.

The 'hash' table is defined thus:

mb_opensuse2=# \d hash
                          Table "public.hash"
    Column     |         Type         | Collation | Nullable | Default 
---------------+----------------------+-----------+----------+---------
 file_id       | integer              |           | not null | 
 mtime         | integer              |           | not null | 
 size          | bigint               |           | not null | 
 md5           | bytea                |           | not null | 
 sha1          | bytea                |           | not null | 
 sha256        | bytea                |           | not null | 
 sha1piecesize | integer              |           | not null | 
 sha1pieces    | bytea                |           | not null | 
 btih          | bytea                |           | not null | 
 pgp           | text                 |           | not null | 
 zblocksize    | smallint             |           | not null | 
 zhashlens     | character varying(8) |           |          | 
 zsums         | bytea                |           | not null | 

I'm no postgresql expert, but 'string' seems to mean fields 'pgp' or 'zhashlens'.

I added some debug output to /usr/lib64/python2.7/site-packages/mb/hashes.py to print the values of pgp and zhashlens.

Hashing '/srv/ftp-stage/pub/opensuse/repositories/home:/legoktm/Debian_9.0/mediawiki_1.31.1.orig.tar.gz'... done.
pgp = '\x88]\x04\x00\x11\x02\x00\x1d\x16!\x04\x1d\x98\x86~\x82\x98,\x8f\xe0\xab\xc2_\x9bi\xb3\x10\x9d;\xb7\xb0\x05\x02[\xa3\xf0\xf8\x00\n\t\x10\x9bi\xb3\x10\x9d;\xb7\xb0@_\x00\x9ft\x82j\x16\xd5\xa0\xb6\xb2\x9f<+,\xe0-\xb7\xac\x9c\xec\x1a\x91\x00\xa0\xae\x93Tg\xe7a\x7f\xd4\x9d#\xcc\xa9\x95DV\xd9*\x12`\xd7'
zhashlens = ''

So, 'pgp' has four 0x00s ... the pgp signature is read from mediawiki_1.31.1.orig.tar.gz.asc, which does indeed contain some gobbledegook.

I ended up renaming:

/srv/ftp-stage/pub/opensuse/repositories/home:/legoktm/xUbuntu_18.04/mediawiki_1.31.1.orig.tar.gz.asc 
to
/srv/ftp-stage/pub/opensuse/repositories/home:/legoktm/xUbuntu_18.04/mediawiki_1.31.1.orig.tar.gz.xasc

and

/srv/ftp-stage/pub/opensuse/repositories/home:/legoktm/Debian_9.0/mediawiki_1.31.1.orig.tar.gz.asc 
to
/srv/ftp-stage/pub/opensuse/repositories/home:/legoktm/Debian_9.0/mediawiki_1.31.1.orig.tar.gz.xasc

After that, the makehashes run continued much longer, but still eventually failed:

Unlinking obsolete '/srv/metalink-hashes/srv/ftp/pub/opensuse/repositories/vdr/openSUSE_Factory_ARM/src/vdradmin-am-3.6.10-4.26.src.rpm.size_627000'
Traceback (most recent call last):
  File "/usr/bin/mb", line 1729, in <module>
    r = mirrordoctor.main()
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 261, in main
    return self.cmd(args)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 284, in cmd
    retval = self.onecmd(argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 422, in onecmd
    return self._dispatch_cmd(handler, argv)
  File "/usr/lib/python2.7/site-packages/cmdln.py", line 1123, in _dispatch_cmd
    return handler(argv[0], opts, *args)
  File "/usr/bin/mb", line 1046, in do_makehashes
    os.makedirs(dst_dir, mode = 0755)
  File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/srv/metalink-hashes/srv/ftp/pub/opensuse/repositories/vdr/openSUSE_Leap_15.3/noarch'

The following files contain weird binary gunk:

./repositories/home:/jejb1:/TPM/Debian_Testing/gnupg2_2.2.25.orig.tar.bz2.asc
./repositories/home:/legoktm:/mediawiki/Debian_9.0/mediawiki_1.31.1.orig.tar.gz.asc
./repositories/home:/legoktm:/mediawiki/Raspbian_9.0/mediawiki_1.31.1.orig.tar.gz.asc
./repositories/home:/legoktm:/mediawiki/xUbuntu_18.04/mediawiki_1.31.1.orig.tar.gz.asc

I think I ought to be able to exclude those using the '-i' option on the makehashes command.

pjessen wrote:

I think I ought to be able to exclude those using the '-i' option on the makehashes command.

1st attempt - using two '-i' options:

/usr/bin/mb makehashes -b /srv/ftp-stage/pub/opensuse -t /srv/metalink-hashes/srv/ftp/pub/opensuse/ -i 'repositories/home:/legoktm:/mediawiki/Debian_9.0/mediawiki_1.31.1' -i 'repositories/home:/jejb1:/TPM/Debian_Testing/gnupg2_2.2.25'

Looks like the command doesn't work with multiple exclusions ....

Combining the two regexes into one:

-i 'repositories/home:/legoktm:/mediawiki/Debian_9.0/mediawiki_1.31.1|repositories/home:/jejb1:/TPM/Debian_Testing/gnupg2_2.2.25'

The shell somehow sees the '|' as a pipe ... well, that is probably due to the makehashes command being wrapped in another script (withlock).
Running it manually seems to work, but I still need to run it with cron and withlock.

Better "solution" - hack /usr/lib64/python2.7/site-packages/mb/hashes.py :

    from string import printable
    if set(str(self.hb.pgp)).difference(printable):
        self.hb.pgp=''

It seems reasonable to ignore invalid characters. Running a new makehashes just now, we'll see if it works.

It worked! makehashes is running continually, see /home/mirror/bin/update_metalinks2 - doing repositories takes quite long though, the last run took 37hours ... another one started last night at 1916, it is still running.

We are currently creating hashes for

repositories distribution debug "history" "source" update factory-snapshot

"factory-snapshot" does not exist any more, we have "tumbleweed". Also, "ports" is missing. I've updated /home/mirror/bin/update_metalinks2 accordingly:

repositories distribution debug "history" "source" update tumbleweed ports

I worry about the time needed to hash "repositories", if it is going to continue to take so long, we'll have to think of parallelising this process.

pjessen wrote:

It worked! makehashes is running continually, see /home/mirror/bin/update_metalinks2 - doing repositories takes quite long though, the last run took 37hours ... another one started last night at 1916, it is still running.

Started at 2021-09-13T19:16:59, now at 2021-09-15T06:24:51 it is still running. The previous run took exactly 37h:37m:16s, so maybe this one will finish around 08:53 UTC?

pjessen wrote:

Started at 2021-09-13T19:16:59, now at 2021-09-15T06:24:51 it is still running. The previous run took exactly 37h:37m:16s, so maybe this one will finish around 08:53 UTC?

Thu 16 Sep 09:09:00 Still running, almost 60 hours later.

Now at 85 hours. This is getting ridiculous - the /srv filesystem is also 98% full ....

It finally finished, this morning:

Start time  = 2021-09-13T19:16:59
End time    = 2021-09-18T06:06:50
Elapse time = 106h 49m 51s

For now, I have removed 'repositories' from the directory list in /home/mirror/bin/update_metalinks2, I think I'll let it run in parallel to the rest.

Mostly notes for myself -

running the makehashes on 'distribution debug history source update tumbleweed ports' can now be done in reasonable time, maybe 1-2 hours in total. 'repositories' still takes a day, sometimes more.
I have created a metalinkhashes@.service - if instance='none', it runs the directories above.

I have enabled two services running continually -

metalinkhashes@none.service
metalinkhashes@repositories.service

Output is sent to 'admin-auto', but only when there is anything to report.
metalinkhashes@none.service could really be run once a day on a cron or timer, but with the rate of change we are seeing, I think it is a good idea to let it run continually.

I have created a separate service for 'ports', I noticed the makehashes run sometimes taking quite a while. Otherwise, my hack (comment 10) worked just fine.
I guess I need to get it merged into our mirrorbrain package.