openSUSE admin

This tracks implementation of a new S3 compatible storage setup.

Currently we have a single Minio instance on minio.i.o.o, acting as a backend for paste.o.o, which lacks redundancy and is not well suited for serving additional applications.

I currently have two more use cases for S3:

• DNS (replication of zones via LightningStream)
• Monitoring (data storage for Thanos)

And there might be more use cases in the future, as object storage is popular with various web apps.

It would not be wise to have these various use cases rely on the single existing instance of Minio.

For a new, improved, S3 setup, I have the following idea:

• 6x VMs (3 in PRG2, 3 in SLC1)
• each VM with a SSD disk for metadata, and a bigger HDD disk for data blobs
• each VM with an installation of Garage
• Garage to be configured with a replication factor of 5 (or 3? to be investigated)
• our existing reverse proxy pair can act as a frontend (alternatively a Garage "gateway" could be used, having the benefit of the traffic going directly to the backend node storing the data instead of needing to be forwarded, but it might not be needed for our use case and can be added later if performance improvement is desired)

Considerations which went into this design idea:

• Garage is built with geographic replication in mind
• Garage is simple to maintain and has good documentation
• Garage is in Factory ;-)
• The chosen replication factor will transparently compensate for failure of either:
  • all nodes in one site, and data will still be consistent in the other site
  • failure of up to one node in each individual site
• Geographic replication will allow us to deploy geographically redundant web services (at least for applications which don't require any other site-specific backends)