Project

General

Profile

Actions

tickets #36766

closed

Tumbleweed Snapshots hosting

Added by jberry almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Mirrors
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:

Description

After my presentation [1] at the openSUSE conference it was suggested I renew
efforts for official hosting. I see three possible routes:

1) hosted on pontifex (download.o.o) with mirror brain integration
2) hosted on pontifex without mirror brain integration
3) hosted on separate machine

Given that roughly 9% of the raw size is used for 50 snapshots a large portion
of the files are available on the live mirrors since they do not change. As
such bandwidth could be saved by using the existing mirror network.

The snapshots can be provided via rsync module for mirrors that opt in to host
them as well (or in addition to mirror brain for existing mirrors).

Short of someone with knowledge of that setup and interest in making this
happen I would suggest #3 to at minimum provide official hosting in the
simplest form. Things can be improved as necessary.

The second point arises which is what format to utilize.

1) hardlinks
2) HTTP redirects like S3

The code for making snapshots can be found at boombatower/tumbleweed-snapshot
[2]. The hardlink approach being available in hardlink branch (same as 0.2.0
release tag).

The existing snapshots should be usable by syncing from AWS S3 to whichever
machine we choose.

Preferably, enough storage space could be provided for debug repo in addition
to oss and non-oss repos. The current storage consumption for 50 snapshots can
be seen in metrics.o.o [3] as just over 300GB. 500GB should be sufficient and
1TB enough for debug repo. I have a copy of debug from a while back which is
just under 60GB (or the same as oss and non-oss combined).

58G /srv/www/htdocs/opensuse/debug/

As such even 500GB could suffice, especially if a) less snapshots are stored
or, b) drop debug after a shorter interval.

I see the following on pontifex currently:

/dev/mapper/pontifex3-pontifex3 13T 12T 1.5T 90% /srv

Last I looked there was 0.5T available. I am not sure how much this fluctuates
which is a concern for use more space. There is already a package for the
original code (which should likely be updated for more recent source).

If #3 is done would that mean a domain apart form download.o.o?

I am happy to perform #3 and alternatively #1 or #2 with direction or help
someone else set things up.

Once everything is said and done, I can update the tumbleweed-cli to point to
the new hosting location.

[1] https://www.youtube.com/watch?v=CRszp1p47BM
[2] https://github.com/boombatower/tumbleweed-snapshot
[3] https://metrics.opensuse.org/d/osrt_tumbleweed_snapshots/tumbleweed-snapshots?panelId=2&fullscreen&orgId=1
[4] https://build.opensuse.org/package/show/openSUSE:infrastructure/
tumbleweed-snapshot

--
Jimmy

Actions #1

Updated by jberry almost 6 years ago

  • Private changed from Yes to No
Actions #2

Updated by tampakrap almost 6 years ago

  • Assignee set to tampakrap
Actions #3

Updated by jberry over 5 years ago

If possible could we just increase the disk on pontifex2 (download.o.o) and host there? Less setup and one less machine to manage for doing the same things.

After reviewing rsync setup it should be safe to place the snapshots in: /srv/ftp/pub/opensuse/tumbleweed-snapshot.

This can then be aliased in the main apache _download.conf as either:

My preference being the first one. Since the load is significantly less and the latest snapshot is served from live mirrors it should be fine to serve from downloadcontent.opensuse.org which likely will work automatically as MirrorBrainFallback is configured to that. Otherwise can be explicitly excluded if necessary.

Keeping outside of opensuse/tumbleweed should exclude it from the normal rsync modules and not being in /srv/ftp-stage/pub it will not be in the opensuse-full module anyway. If desired can always setup a module for this as well.

Depending on opinions this can still be done via hard-links or redirects. Is it feasible to ensure 1TB or at least 500GB of space for snapshots?

Actions #4

Updated by pjessen over 5 years ago

  • Category set to Mirrors

On pontifex, I see /etc/nginx/vhosts.d/001-tumbleweed-snapshots.conf-20171206
which has a host named 'twsnapshot.opensuse.org' ?

I don't know where/how to get the needed diskspace. I expect we won't have many sites mirroring the snapshots, maybe it's better to leave Mirrorbrain out of it and just set it up in nginx? Similar to the above, but under http://download.o.o/tumbleweed/snapshot ?

Actions #5

Updated by jberry over 5 years ago

Not doing anything with mirror brain is fine with me and what I was suggesting in last comment. I would rather using main domain for serving so it shows up logs and can thus be attributed to Tumbleweed usage (since it is). The vhost file referenced was created by someone else prior to my access to the machine.

If I understand you correctly, we agree on the path forward just need to disk allotment.

Actions #6

Updated by jberry over 5 years ago

  • Assignee changed from tampakrap to jberry

Per e-mail discussion I will be deploying this in a limited form until disk requirements are worked out.

After installing on pontifex (will do out of my branch if necessary) the following config will be used (/etc/tumbleweed-snapshot.conf):

SOURCE_DIR="/srv/ftp/pub/opensuse"
SNAPSHOT_DIR="/srv/ftp/pub/opensuse/tumbleweed/history"
SNAPSHOT_COUNT_MAX=5
RSYNC_INCLUDE="/usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt"

Seems like no need to touch stage since only want to update once actually published and will redirect to live while copying anyway.

/usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt will by a copy of the default include-tumbleweed.txt with the debug and source inclusions commented out.

The tumbleweed-snapshot user will be created and given write access to $SNAPSHOT_DIR.

I'll need to review the rsync modules to decide what changes need to be made there, but will include them here.

Finally the tumbleweed-snapshot.service will be triggered hourly by a tumbleweed-snapshot.timer which will check for a newly published tree (reads two files) which should keep it from being tangled up in the existing publishing scripts.

Actions #7

Updated by jberry over 5 years ago

It may also make sense to use:

SNAPSHOT_DIR="/srv/ftp/pub/opensuse/history"

since the file structure is set to include the tree under opensuse (ie. tumbleweed/repo/oss) since it can mirror the debug and source paths down the road.

Actions #8

Updated by jberry over 5 years ago

Likely also makes sense to migrate the tumbleweed-{cli,snapshot} repositories under the openSUSE namespace, but no rush and still need to increase the number of snapshots.

Actions #9

Updated by jberry over 5 years ago

Made the following changes:

sudo zypper ar -p 95 https://download.opensuse.org/repositories/home:/boombatower:/branches:/openSUSE:/infrastructure/openSUSE_Leap_42.3/ boombatower-tumbleweed-snapshot
sudo zypper ref
sudo zypper in tumbleweed-snapshot
sudo vim /etc/tumbleweed-snapshot.conf

SOURCE_DIR="/srv/ftp/pub/opensuse"
SNAPSHOT_DIR="/srv/ftp/pub/opensuse/history"
SNAPSHOT_COUNT_MAX=10
RSYNC_INCLUDE="/usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt"

sudo cp /usr/share/tumbleweed-snapshot/rsync/include-tumbleweed.txt /usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt

# commented debug and source

sudo mkdir /srv/ftp/pub/opensuse/history
sudo chown -R tumbleweed-snapshot:tumbleweed-snapshot /srv/ftp/pub/opensuse/history


sudo vim /etc/rsyncd-stage.conf

# jberry@suse.com @ 2018-08-29: for boombatower/tumbleweed-snapshot
[opensuse-history]
        path = /srv/ftp/pub/opensuse/history
        comment = Historical snapshots of a subset of repositories and content.
        list = yes
        uid = nobody
        gid = nogroup
        dont compress = *

sudo systemctl start tumbleweed-snapshot

Verified new module (opensuse-history) works on stage.opensuse.org. First snapshot creation (longest) is running and will enable timer if all looks good. Public facing URL http://download.opensuse.org/history/.

Actions #10

Updated by jberry over 5 years ago

First snapshot completed and verified using docker container that rpm-md data is the same and rpms are installable.

sudo systemctl enable tumbleweed-snapshot.timer
sudo systemctl start tumbleweed-snapshot.timer

The .htaccess file to redirect current snapshot to normal paths does not seem to work. Assuming this has to do with apache configuration relating to reading those files. Any opinions on the matter? It works as is, but would be more bandwidth efficient to redirect.

Actions #11

Updated by jberry over 5 years ago

  • Assignee changed from jberry to tampakrap

Done from my end short of the .htaccess decision and config tweak.

Assigning back to tampakrap for disk increase after which /etc/tumbleweed-snapshot.conf should increase SNAPSHOT_COUNT_MAX to 50 or somesuch. Can also uncomment debug in /usr/share/tumbleweed-snapshot/rsync/include-tumbleweed-only-repo.txt if we have space to include that as that would be really nice.

Actions #12

Updated by jberry over 5 years ago

I'll observe and review the snapshots created over the next couple weeks to make sure everything is working properly. I won't announce or recommend usage until we have a full set of snapshots and looks good. I'll keep my AWS-S3 copy around until a) we serve more than 10, and b) includes a complete set of that number.

Actions #13

Updated by jberry over 5 years ago

Created and deployed related disk metrics (for the same data provided by AWS cloudwatch).

https://github.com/openSUSE/openSUSE-release-tools/pull/1678
https://metrics.opensuse.org/d/osrt_history/osrt-history?orgId=1

Actions #14

Updated by lnussel over 5 years ago

the files need to be put in /srv/ftp-stage for two reasons

  • ~mirror/bin/publish_opensuse would delete anything in /srv/ftp that is not on ftp-stage (unless one adjusts publish_opensuse)
  • mirrors won't pick up the new location automatically. You can adjust the include/exclude lists of the existing modules to only include the history in the larger modules.

The directory can be bind mounted from /srv/ftp-stage to /srv/ftp to make the files visible immediately.

Actions #15

Updated by jberry over 5 years ago

Complete bind mount:

sudo mkdir /srv/ftp-stage/pub/opensuse/history
sudo chown -R tumbleweed-snapshot:tumbleweed-snapshot /srv/ftp-stage/pub/opensuse/history
sudo mount --bind /srv/ftp/pub/opensuse/history/ /srv/ftp-stage/pub/opensuse/history/
Actions #16

Updated by jberry over 5 years ago

Added line to /etc/fstab:

/srv/ftp/pub/opensuse/history             /srv/ftp-stage/pub/opensuse/history           auto  bind                                                      0 0
Actions #17

Updated by jberry over 5 years ago

Based on the contents of ~mirror/bin/publish_factory_leap it would seem withlock -q /home/mirror/var/run/withlock/LOCK-publish would be active whenever tumbleweed is being published. As such it would likely make sense to check for the existence of that lock file and skip snapshot run until it no longer exists?

For my S3 workaround I delay 10 hours to wait for rsync mirrors to catch up (redirect to current version active right away so usable by end-users). I was concerned about catching the file system in inconsistent state mid sync and was debating adding a delay mechanism, but look for lock is more direct although it then relies on this setup. Not sure which is better since a delay is never guaranteed.

Actions #18

Updated by jberry over 5 years ago

Update vhost config to allow for .htaccess to set RedirectMatch from /history directory.

$ sudo cp /etc/apache2/vhosts.d/_download.conf /etc/apache2/vhosts.d/_download.conf.20180906
$ sudo vim /etc/apache2/vhosts.d/_download.conf
$ diff -u /etc/apache2/vhosts.d/_download.conf.20180906 /etc/apache2/vhosts.d/_download.conf
--- /etc/apache2/vhosts.d/_download.conf.20180906       2018-09-06 20:03:39.437752401 +0000
+++ /etc/apache2/vhosts.d/_download.conf        2018-09-06 20:07:56.640943554 +0000
@@ -181,6 +181,11 @@
       MirrorBrainFallback eu de http://downloadcontent.opensuse.org/
     </Directory>

+    # poo#36766: allow .htacess override to redirect current snapshot to root
+    <Directory /srv/ftp/pub/opensuse/history>
+      AllowOverride FileInfo
+    </Directory>
+
     Use repomdfiles_nocache

     RewriteEngine On
$ sudo rcapache2 reload

Assuming no one indicates my understanding of lock file is wrong I'll add that option to tumbleweed snapshot tool, deploy, and configure it. After which it would seem everything is completed short of increasing the number of snapshots which is disk and time dependent.

Currently:

$ cat /srv/ftp/pub/opensuse/history/20180903/.htaccess
# Placed in sync directory to redirect request for current snapshot resources to
# main mirror network and thus only server request for older snapshots.
RedirectMatch "/\d{8}/(.*)" "http://download.opensuse.org/$1"

Which means that any files accessed within that snapshot will be redirected to root and normal mirror network.

While older snapshots are not redirect and served by whatever mirrors have them (none currently).

Actions #19

Updated by lnussel over 5 years ago

jberry wrote:

Based on the contents of ~mirror/bin/publish_factory_leap it would seem withlock -q /home/mirror/var/run/withlock/LOCK-publish would be active whenever tumbleweed is being published. As such it would likely make sense to check for the existence of that lock file and skip snapshot run until it no longer exists?

The lockfile only exists while the script runs. That doesn't tell whether or not the script publishes TW

Actions #20

Updated by lnussel over 5 years ago

jberry wrote:

Update vhost config to allow for .htaccess to set RedirectMatch from /history directory.

Could we use the .current.txt file file instead? (RewriteMap tw_iso_current). That would avoid .htaccess outside the main config.

Actions #21

Updated by lnussel over 5 years ago

jberry wrote:

sudo chown -R tumbleweed-snapshot:tumbleweed-snapshot /srv/ftp-stage/pub/opensuse/history

Any reason not to use mirror:stage?

Actions #22

Updated by jberry over 5 years ago

The lockfile only exists while the script runs. That doesn't tell whether or not the script publishes TW

The snapshot tool is only interested in the underlying file tree not being in flux. Not sure what's best to use.

Could we use the .current.txt file file instead? (RewriteMap tw_iso_current). That would avoid .htaccess outside the main config.

Potentially, but I would need to explore that approach.

Any reason not to use mirror:stage?

Just sticking with the same permissions on source directory. Those were used to avoid the tool messing up anything else as had been discussed almost 2 years ago when the tool was originally packaged.

Actions #23

Updated by jberry over 5 years ago

Could we use the .current.txt file file instead? (RewriteMap tw_iso_current). That would avoid .htaccess outside the main config.

The documentation (and examples) seem to indicate that is for mapping the a matched path to output path which would mean it would have to match something like all history/.* paths and then use some sort of map to map current to root and everything else to same directory. Not sure that really fits our use-case and feels a bit awkward. Either way from what I gather a bit of file manipulation would need to be done to generate such a map.

What we really want is a custom matching pattern as the destination path pattern is always the same. I'd have to dig around Apache docs to figure out if/how to do something like that. Not hugely of interest unless compelling reason.

Updated tumbleweed-snapshot tool to provide SOURCE_LOCK option, configured in /etc/tumbleweed-snapshot.conf to be /home/mirror/var/run/withlock/LOCK-publish as that should be sufficient to block whenever scripts are potentially messing with directory, and updated to version containing option.

This is everything relevant to resolve short of disk space increase.

Actions #24

Updated by jberry over 5 years ago

Updated /etc/mirrorbrain.conf to include history in each scan_top_include directive.

Actions #25

Updated by jberry over 5 years ago

Announced to mirror lists (https://lists.opensuse.org/mirror/2018-09/msg00002.html) and looks like some already have it so we'll see if mirrorbrain changes take.

Actions #26

Updated by jberry over 5 years ago

Setup a gitlab pipeline (https://gitlab.suse.de/jberry/mirrorbrain-check/pipelines) that will fail if redirected to a mirror when accessing /history content and will e-mail me. Hopefully, this change will work and just takes some time for scanner to cycle.

Actions #27

Updated by jberry over 5 years ago

Added history to for list in /home/mirror/bin/update_metalinks. Seems like the best place since not part of /home/mirror/publish_distro/publish_distro since history is not really a distro and may eventually include more than TW. Dug through /etc/cron.d/mirrorstuff and looked at process list and this seems like the missing bit as the hash need to be there for the scanner to utilize (or so it seems). Hopefully that will create directory in /srv/metalink-hashes/srv/ftp/pub/opensuse/ and then find out if need to do more to get scanner to work.

Actions #28

Updated by jberry over 5 years ago

The hashes were generated, but still doesn't work guessing due to scan not running. Not exactly clear what runs scans outside of the repository pushing and such.

Actions #29

Updated by lnussel over 5 years ago

Theo, isn't there separate scanner host? Does it need to be configured?

Actions #30

Updated by jberry over 5 years ago

I scan mirrorbrain scan jobs run, but they are tied to distro pushes which I see the script triggers and general repository updates. Perhaps I need to make a new cron job for the history stuff as it does not seem to fit anywhere else.

Actions #31

Updated by jberry over 5 years ago

Invoking manually just to see if this is the trick. If it works I'll go ahead and add to crontab unless someone suggests something better.

sudo -u repopusher mb scan -a -j 4 -d history
Actions #32

Updated by jberry over 5 years ago

Alternatively, perhaps the best route is to set it up to scan a specific snapshot directory once a new one is published. Likely n-1 is always best to give time to mirror. Since old ones shouldn't change and just waste time scanning.

Actions #33

Updated by jberry over 5 years ago

The manual scan seems to have done the trick.

The following redirects to a mirror http://download.opensuse.org/history/20180919/tumbleweed/repo/oss/GPLv3.txt.

Actions #34

Updated by cboltz over 5 years ago

Alternatively, perhaps the best route is to set it up to scan a specific snapshot directory once a new one is published. Likely n-1 is always best to give time to mirror. Since old ones shouldn't change and just waste time scanning.

I'm not a mirrorbrain expert, but nevertheless - I'm not sure if this is a good idea ;-)

Mirrors might delete old snapshots (maybe some choose to host less snapshots than download.o.o), and mirrorbrain needs to know that so that it no longer redirects there.

Also, knowing which mirrors already have the latest snapshot can't hurt ;-)

Actions #35

Updated by jberry over 5 years ago

Yeah, I just figured it's a lot of files to check on a regular basis (currently over 500k), but I agree with your points. As such perhaps the best compromise is a semi-slow refresh interval like 6-18 hours. I'm open to suggestions.

I am also not to keen on setting up a new lock, log, and log rotation for this. I would lean towards adding a systemd timer and service which should accomplish the equivalent.

Actions #36

Updated by jberry over 5 years ago

Completed in 6.2 hours

Obviously with a higher concurrency value it would complete faster.

Actions #37

Updated by jberry over 5 years ago

Created the following service and timer.

/usr/lib/systemd/system/mirrorbrain-history.service:

[Unit]
Description=MirrorBrain scan for history directory

[Service]
User=repopusher
SyslogIdentifier=mirrorbrain
ExecStart=/usr/bin/mb scan -a -j 8 -d history
RuntimeMaxSec=24 hour

[Install]
WantedBy=multi-user.target

/usr/lib/systemd/system/mirrorbrain-history.timer:

[Unit]
Description=MirrorBrain scan for history directory

[Timer]
OnBootSec=120
OnUnitInactiveSec=12 hour
Unit=mirrorbrain-history.service

[Install]
WantedBy=timers.target
Actions #38

Updated by jberry over 5 years ago

  • Status changed from New to Resolved
  • Assignee changed from tampakrap to jberry

Forgot to explicitly mention I enabled and started the timer.

Also at this point I believe this is all the desired action items short of increased disk which likely should be broken out as a separate issue. As I do not expect any movement in near future I'll just mark this issue as fixed. Having 15-20 I think is ideal when Tumbleweed is rolling well (near daily) to allow for >= 3 weeks.

Actions #39

Updated by cboltz over 5 years ago

  • Status changed from Resolved to Workable

Created the following service and timer.
/usr/lib/systemd/system/mirrorbrain-history.service:
/usr/lib/systemd/system/mirrorbrain-history.timer:

Only packaged files should go to /usr/lib/systemd/... - and I'd guess you didn't package them ;-) so hiding them in /usr/lib/systemd/ is a bad idea.

/etc/systemd/... is a much better place ;-)

The perfect solution would be to manage them via salt, but since the download.o.o setup sadly was done manually, I won't force you to put the mirrorbrain-history files into salt. Nevertheless, it would be a good start ;-)

Actions #40

Updated by cboltz over 5 years ago

  • Status changed from Workable to In Progress
Actions #41

Updated by jberry over 5 years ago

  • Status changed from In Progress to Resolved

The snapshot service is packaged and everything else I deploy is packaged via openSUSE-release-tools, but this is supplemental to MirrorBrain setup on this box and thus does not fit. I went ahead and moved them to /etc/systemd.

Actions #42

Updated by cboltz over 5 years ago

Thanks!

Actions

Also available in: Atom PDF