openSUSE admin

If possible could we just increase the disk on pontifex2 (download.o.o) and host there? Less setup and one less machine to manage for doing the same things.

After reviewing rsync setup it should be safe to place the snapshots in: /srv/ftp/pub/opensuse/tumbleweed-snapshot.

This can then be aliased in the main apache _download.conf as either:

• http://download.opensuse.org/tumbleweed/snapshot/
• http://download.opensuse.org/tumbleweed-snapshot/

My preference being the first one. Since the load is significantly less and the latest snapshot is served from live mirrors it should be fine to serve from downloadcontent.opensuse.org which likely will work automatically as MirrorBrainFallback is configured to that. Otherwise can be explicitly excluded if necessary.

Keeping outside of opensuse/tumbleweed should exclude it from the normal rsync modules and not being in /srv/ftp-stage/pub it will not be in the opensuse-full module anyway. If desired can always setup a module for this as well.

Depending on opinions this can still be done via hard-links or redirects. Is it feasible to ensure 1TB or at least 500GB of space for snapshots?

Not doing anything with mirror brain is fine with me and what I was suggesting in last comment. I would rather using main domain for serving so it shows up logs and can thus be attributed to Tumbleweed usage (since it is). The vhost file referenced was created by someone else prior to my access to the machine.

If I understand you correctly, we agree on the path forward just need to disk allotment.

It may also make sense to use:

SNAPSHOT_DIR="/srv/ftp/pub/opensuse/history"

since the file structure is set to include the tree under opensuse (ie. tumbleweed/repo/oss) since it can mirror the debug and source paths down the road.

Likely also makes sense to migrate the tumbleweed-{cli,snapshot} repositories under the openSUSE namespace, but no rush and still need to increase the number of snapshots.

Made the following changes:

sudo zypper ar -p 95 https://download.opensuse.org/repositories/home:/boombatower:/branches:/openSUSE:/infrastructure/openSUSE_Leap_42.3/ boombatower-tumbleweed-snapshot
sudo zypper ref
sudo zypper in tumbleweed-snapshot
sudo vim /etc/tumbleweed-snapshot.conf

SOURCE_DIR="/srv/ftp/pub/opensuse"
SNAPSHOT_DIR="/srv/ftp/pub/opensuse/history"
SNAPSHOT_COUNT_MAX=10
RSYNC_INCLUDE="/usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt"

sudo cp /usr/share/tumbleweed-snapshot/rsync/include-tumbleweed.txt /usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt

# commented debug and source

sudo mkdir /srv/ftp/pub/opensuse/history
sudo chown -R tumbleweed-snapshot:tumbleweed-snapshot /srv/ftp/pub/opensuse/history


sudo vim /etc/rsyncd-stage.conf

# jberry@suse.com @ 2018-08-29: for boombatower/tumbleweed-snapshot
[opensuse-history]
        path = /srv/ftp/pub/opensuse/history
        comment = Historical snapshots of a subset of repositories and content.
        list = yes
        uid = nobody
        gid = nogroup
        dont compress = *

sudo systemctl start tumbleweed-snapshot

Verified new module (opensuse-history) works on stage.opensuse.org. First snapshot creation (longest) is running and will enable timer if all looks good. Public facing URL http://download.opensuse.org/history/.

First snapshot completed and verified using docker container that rpm-md data is the same and rpms are installable.

sudo systemctl enable tumbleweed-snapshot.timer
sudo systemctl start tumbleweed-snapshot.timer

The .htaccess file to redirect current snapshot to normal paths does not seem to work. Assuming this has to do with apache configuration relating to reading those files. Any opinions on the matter? It works as is, but would be more bandwidth efficient to redirect.

I'll observe and review the snapshots created over the next couple weeks to make sure everything is working properly. I won't announce or recommend usage until we have a full set of snapshots and looks good. I'll keep my AWS-S3 copy around until a) we serve more than 10, and b) includes a complete set of that number.

Created and deployed related disk metrics (for the same data provided by AWS cloudwatch).

https://github.com/openSUSE/openSUSE-release-tools/pull/1678
https://metrics.opensuse.org/d/osrt_history/osrt-history?orgId=1

the files need to be put in /srv/ftp-stage for two reasons

• ~mirror/bin/publish_opensuse would delete anything in /srv/ftp that is not on ftp-stage (unless one adjusts publish_opensuse)
• mirrors won't pick up the new location automatically. You can adjust the include/exclude lists of the existing modules to only include the history in the larger modules.

The directory can be bind mounted from /srv/ftp-stage to /srv/ftp to make the files visible immediately.

Complete bind mount:

sudo mkdir /srv/ftp-stage/pub/opensuse/history
sudo chown -R tumbleweed-snapshot:tumbleweed-snapshot /srv/ftp-stage/pub/opensuse/history
sudo mount --bind /srv/ftp/pub/opensuse/history/ /srv/ftp-stage/pub/opensuse/history/

Added line to /etc/fstab:

/srv/ftp/pub/opensuse/history             /srv/ftp-stage/pub/opensuse/history           auto  bind                                                      0 0

Based on the contents of ~mirror/bin/publish_factory_leap it would seem withlock -q /home/mirror/var/run/withlock/LOCK-publish would be active whenever tumbleweed is being published. As such it would likely make sense to check for the existence of that lock file and skip snapshot run until it no longer exists?

For my S3 workaround I delay 10 hours to wait for rsync mirrors to catch up (redirect to current version active right away so usable by end-users). I was concerned about catching the file system in inconsistent state mid sync and was debating adding a delay mechanism, but look for lock is more direct although it then relies on this setup. Not sure which is better since a delay is never guaranteed.

Update vhost config to allow for .htaccess to set RedirectMatch from /history directory.

$ sudo cp /etc/apache2/vhosts.d/_download.conf /etc/apache2/vhosts.d/_download.conf.20180906
$ sudo vim /etc/apache2/vhosts.d/_download.conf
$ diff -u /etc/apache2/vhosts.d/_download.conf.20180906 /etc/apache2/vhosts.d/_download.conf
--- /etc/apache2/vhosts.d/_download.conf.20180906       2018-09-06 20:03:39.437752401 +0000
+++ /etc/apache2/vhosts.d/_download.conf        2018-09-06 20:07:56.640943554 +0000
@@ -181,6 +181,11 @@
       MirrorBrainFallback eu de http://downloadcontent.opensuse.org/
     </Directory>

+    # poo#36766: allow .htacess override to redirect current snapshot to root
+    <Directory /srv/ftp/pub/opensuse/history>
+      AllowOverride FileInfo
+    </Directory>
+
     Use repomdfiles_nocache

     RewriteEngine On
$ sudo rcapache2 reload

Assuming no one indicates my understanding of lock file is wrong I'll add that option to tumbleweed snapshot tool, deploy, and configure it. After which it would seem everything is completed short of increasing the number of snapshots which is disk and time dependent.

Currently:

$ cat /srv/ftp/pub/opensuse/history/20180903/.htaccess
# Placed in sync directory to redirect request for current snapshot resources to
# main mirror network and thus only server request for older snapshots.
RedirectMatch "/\d{8}/(.*)" "http://download.opensuse.org/$1"

Which means that any files accessed within that snapshot will be redirected to root and normal mirror network.

• http://download.opensuse.org/history/20180903/tumbleweed/repo/oss/GPLv3.txt
• http://mirror.dal10.us.leaseweb.net/opensuse/tumbleweed/repo/oss/GPLv3.txt

While older snapshots are not redirect and served by whatever mirrors have them (none currently).

jberry wrote:

Based on the contents of ~mirror/bin/publish_factory_leap it would seem withlock -q /home/mirror/var/run/withlock/LOCK-publish would be active whenever tumbleweed is being published. As such it would likely make sense to check for the existence of that lock file and skip snapshot run until it no longer exists?

The lockfile only exists while the script runs. That doesn't tell whether or not the script publishes TW

jberry wrote:

Update vhost config to allow for .htaccess to set RedirectMatch from /history directory.

Could we use the .current.txt file file instead? (RewriteMap tw_iso_current). That would avoid .htaccess outside the main config.

jberry wrote:

sudo chown -R tumbleweed-snapshot:tumbleweed-snapshot /srv/ftp-stage/pub/opensuse/history

Any reason not to use mirror:stage?

The lockfile only exists while the script runs. That doesn't tell whether or not the script publishes TW

The snapshot tool is only interested in the underlying file tree not being in flux. Not sure what's best to use.

Could we use the .current.txt file file instead? (RewriteMap tw_iso_current). That would avoid .htaccess outside the main config.

Potentially, but I would need to explore that approach.

Any reason not to use mirror:stage?

Just sticking with the same permissions on source directory. Those were used to avoid the tool messing up anything else as had been discussed almost 2 years ago when the tool was originally packaged.

Could we use the .current.txt file file instead? (RewriteMap tw_iso_current). That would avoid .htaccess outside the main config.

The documentation (and examples) seem to indicate that is for mapping the a matched path to output path which would mean it would have to match something like all history/.* paths and then use some sort of map to map current to root and everything else to same directory. Not sure that really fits our use-case and feels a bit awkward. Either way from what I gather a bit of file manipulation would need to be done to generate such a map.

What we really want is a custom matching pattern as the destination path pattern is always the same. I'd have to dig around Apache docs to figure out if/how to do something like that. Not hugely of interest unless compelling reason.

Updated tumbleweed-snapshot tool to provide SOURCE_LOCK option, configured in /etc/tumbleweed-snapshot.conf to be /home/mirror/var/run/withlock/LOCK-publish as that should be sufficient to block whenever scripts are potentially messing with directory, and updated to version containing option.

This is everything relevant to resolve short of disk space increase.

Updated /etc/mirrorbrain.conf to include history in each scan_top_include directive.

Announced to mirror lists (https://lists.opensuse.org/mirror/2018-09/msg00002.html) and looks like some already have it so we'll see if mirrorbrain changes take.

Setup a gitlab pipeline (https://gitlab.suse.de/jberry/mirrorbrain-check/pipelines) that will fail if redirected to a mirror when accessing /history content and will e-mail me. Hopefully, this change will work and just takes some time for scanner to cycle.

Added history to for list in /home/mirror/bin/update_metalinks. Seems like the best place since not part of /home/mirror/publish_distro/publish_distro since history is not really a distro and may eventually include more than TW. Dug through /etc/cron.d/mirrorstuff and looked at process list and this seems like the missing bit as the hash need to be there for the scanner to utilize (or so it seems). Hopefully that will create directory in /srv/metalink-hashes/srv/ftp/pub/opensuse/ and then find out if need to do more to get scanner to work.

The hashes were generated, but still doesn't work guessing due to scan not running. Not exactly clear what runs scans outside of the repository pushing and such.

Theo, isn't there separate scanner host? Does it need to be configured?

I scan mirrorbrain scan jobs run, but they are tied to distro pushes which I see the script triggers and general repository updates. Perhaps I need to make a new cron job for the history stuff as it does not seem to fit anywhere else.

Invoking manually just to see if this is the trick. If it works I'll go ahead and add to crontab unless someone suggests something better.

sudo -u repopusher mb scan -a -j 4 -d history

Alternatively, perhaps the best route is to set it up to scan a specific snapshot directory once a new one is published. Likely n-1 is always best to give time to mirror. Since old ones shouldn't change and just waste time scanning.

The manual scan seems to have done the trick.

The following redirects to a mirror http://download.opensuse.org/history/20180919/tumbleweed/repo/oss/GPLv3.txt.

Alternatively, perhaps the best route is to set it up to scan a specific snapshot directory once a new one is published. Likely n-1 is always best to give time to mirror. Since old ones shouldn't change and just waste time scanning.

I'm not a mirrorbrain expert, but nevertheless - I'm not sure if this is a good idea ;-)

Mirrors might delete old snapshots (maybe some choose to host less snapshots than download.o.o), and mirrorbrain needs to know that so that it no longer redirects there.

Also, knowing which mirrors already have the latest snapshot can't hurt ;-)

Yeah, I just figured it's a lot of files to check on a regular basis (currently over 500k), but I agree with your points. As such perhaps the best compromise is a semi-slow refresh interval like 6-18 hours. I'm open to suggestions.

I am also not to keen on setting up a new lock, log, and log rotation for this. I would lean towards adding a systemd timer and service which should accomplish the equivalent.

Completed in 6.2 hours

Obviously with a higher concurrency value it would complete faster.

Created the following service and timer.

/usr/lib/systemd/system/mirrorbrain-history.service:

[Unit]
Description=MirrorBrain scan for history directory

[Service]
User=repopusher
SyslogIdentifier=mirrorbrain
ExecStart=/usr/bin/mb scan -a -j 8 -d history
RuntimeMaxSec=24 hour

[Install]
WantedBy=multi-user.target

/usr/lib/systemd/system/mirrorbrain-history.timer:

[Unit]
Description=MirrorBrain scan for history directory

[Timer]
OnBootSec=120
OnUnitInactiveSec=12 hour
Unit=mirrorbrain-history.service

[Install]
WantedBy=timers.target

Thanks!