openSUSE admin

Current behaviour:

• old files are dropped immediately when new revision is published by obs. This may result in various client errors depending on state of ongoing operations and/or caches of package managers or other tools.
• /history has files which can be shown in /tumbleweed/repo and provide easy access to older versions of .rpm, as well as solve timing issues while new version of TW is being published.

Proposed solution - use 3 locations and hard links to files to keep old versions around for some time:

A. Instead of two locations (ftp and ftp-stage) - use three locations:

• in (where new version is pushed from OBS)
• hist - will contain current version of each repo and previous version(s) - (similar to /history for tumbleweed, just limited to 1 previous version kept for few hours for most of cases).
• live - accumulated files from current and previous revision(s), configurable by number of old versions and/expire TTL for old snapshots (for /history it can be 20 old versions, for other places e.g. single old version with TTL 1 or 2 hours).

live will be shown as content of download.o.o;
hist will be used to manage and retain revisions published by OBS;
in is place where OBS will be publish into.

B. In the simplest straightforward case the mirrors will sync from live.
In an advanced scenario we create "smooth sync" protocol for mirrors, where they sync from hist (taking advantage of hardlinks) and be able to maintain live location on their own.
We keep at least two versions (previous and current) for each repo for some hours.

C. download.opensuse.org takes extra care of /repomd/repodata.xml in live folder and renders previous version of it for few hours after release. (While rsync delivers new version to the mirrors).

Example:
Step 1. all 3 locations have the same state:
/myrepo/noarch/mypackage10.1.rpm
/myrepo/repodata/xx-primary.xml.gz
/myrepo/repodata/repomd.xml (has reference to xx-primary.xml.gz)

Step 2. new version of lib is pushed by OBS, so content of /in becomes:
/in/myrepo/noarch/mypackage11.1.rpm
/in/myrepo/repodata/xy-primary.xml.gz
/in/myrepo/repodata/repomd.xml (has reference to xy-primary.xml.gz)

Step 3. the repo is synked to hist, which now has:
/hist/myrepo/noarch/mypackage11.1.rpm
/hist/myrepo/repodata/xy-primary.xml.gz
/hist/myrepo/repodata/repomd.xml (has reference to xy-primary.xml.gz)
/hist/myrepo#230717/noarch/mypackage11.1.rpm
/hist/myrepo#230717/repodata/xy-primary.xml.gz
/hist/myrepo#230717/repodata/repomd.xml (has reference to xy-primary.xml.gz)
/hist/myrepo#230716/noarch/mypackage10.1.rpm
/hist/myrepo#230716/repodata/xx-primary.xml.gz
/hist/myrepo#230716/repodata/repomd.xml (has reference to xx-primary.xml.gz)

Step 4. live view is updated, having:
/live/myrepo/noarch/mypackage10.1.rpm
/live/myrepo/noarch/mypackage11.1.rpm
/live/myrepo/repodata/xy-primary.xml.gz
/live/myrepo/repodata/xx-primary.xml.gz
/live/myrepo/repodata/repomd.xml (flexible behavior)

Step 5. Few hours passed - we delete /hist/myrepo#230716 and update /live/myrepo/ accordingly.

Prototype:
Tool which manages three locations (in, hist, live) using hard link to save space:
https://github.com/andrii-suse/PubliHist

Prototype for handling /history using PubliHist:
https://github.com/boombatower/tumbleweed-snapshot/compare/master...andrii-suse:tumbleweed-snapshot:PubliHist
Prototypes:
Tool which manages three locations (in, hist, live) using hard link to save space:
https://github.com/andrii-suse/PubliHist

Prototype for handling /history using PubliHist:
https://github.com/boombatower/tumbleweed-snapshot/compare/master...andrii-suse:tumbleweed-snapshot:PubliHist

Thanks for your time Andrii,

This would be a nice improvement. Always when there's a fresh Tumbleweed snapshot hitting the mirrors there will be people in the support rooms asking for help because Zypper is outputting some issue regarding repository metadata.

I know that people aren't supposed to be redirected to outdated mirrors, but I wonder if we are certain that our protocols are working reliably on that front, because from time to time I see people ending up in outdated/troubled mirrors. Of course I still don't know exactly what, aside from download.opensuse.org, are supposed to offer repo metadata yet, do regional MirrorCache instances provide repo metadata too?

I'm afraid I can tell which model from your proposition seems the best, for now, though.