As Ticket #2302 explains, we are currently facing an issue with the scripts that are re-creating the hotstuff-XXXg modules on stage.opensuse.org and rsync.opensuse.org : all directories created by this script are producing empty directories.
The developer of this script is currently on vacation, so we might need to wait until (at least) Monday, 2014-04-14, before this will get fixed.
As result, we stopped the rsync server on rsync.opensuse.org and stage.opensuse.org for now to avoid that people remove 30 - 640 GB of content if they run their rsync commands with the recommended "--delete --delete-after" option.
The database cluster behind http://download.opensuse.org/ has a new home: two machines with 24 CPU Cores and 128 GB RAM.
But the cluster is not running directly on this hardware - instead we have two virtual machines (KVM) that are currently just using 32 GB RAM and 10 CPUs each. This has some interesting benefits:
* Rebooting a virtual machines takes less than one minute. The old, bare metal systems took up to 10 minutes for a reboot. After that, the slave database needs ~5-10 minutes to resync with the master. So a complete "kernel update round" took ~30 minutes only for these two hosts in the past.
* If we need to reboot the underlying systems (the host), all virtual guests can be migrated without any downtime to the other host.
* We have enough free space for the migration of the web servers in front of the database (the ones every user reaches if he visits http://download.opensuse.org/ ) - so this will be the next step
The new machine behind https://openqa.opensuse.org/ has 128 GBB RAM and a 3.6 TB RAID system, allowing to run all tests in tmpfs (RAM) and to store the test results for a very long time.
As always, we used the chance to move the web frontend behind our HA-Proxy, which allows us to provide you immediately with a "service down" page and (hopefully more important) to scale up to many more "backends" once the amount of users trying to see the latest results is increasing over the limit one single machine can handle.
openSUSE admin: Status rsync.opensuse.org (3 comments)
The funny part with RAID5 is: you loose one hard disk in size, but you also can only loose one hard disk in your RAID array. As always, Mr. Murphy knows about this fact and kills two hard disks at once...
So now after replacing two of six hard disks in the system, we are back online (after a complete re-install and re-sync, of course). As the PERC5i RAID controller on this 7 year old machine only allows RAID level 0, 1, 10 and 5, we run again with RAID 5, but this time also defined one of the disks as hot spare - loosing another TB for our data...
openSUSE admin: Bye, bye 11.4 and 12.1 (1 comment)
As you might note by reading https://en.opensuse.org/Lifetime openSUSE currently supports only 12.3 and 13.1. Yes, this means that:
* 12.1 and even
already reached their end of life state and are therefor not supported any more.
As Admins are born to be lazy, we still had some old (namely 11.4, 12.1 and 12.2) repositories and their files on http://download.opensuse.org/ - which might helped users with their old installations.
But as the openSUSE Build Service is gaining more and more popularity, the used space is getting more and more a problem. So today we removed the old ISOs and repositories of 11.4 and 12.1 now from download.opensuse.org - and like to point you to one of the nice mirror admins that still provide the outdated files.
The problem was fixed by
* starting our hot-standby database as new master
* replacing the half broken SSD with a new one
* re-syncing from the current master
...and while we are on it, we took the chance to let PostgreSQL do some cleanups via the "vacuum" and "reindex" commands: the database shrunk from ~30GB to ~10GB, but this took nearly 2 hours!
During the whole time, download.opensuse.org was still reachable - just a bit slow for 1 hour during the vacuuming (sorry for that). Now everything is back to normal, but we will try to figure out how the autovacuum can be tuned, so the database cleanup can be done automatically.
The server hosting http://planet.opensuse.org/global/ was sponsored by a couple of persons during the last years. Special thanks go to Pascal Bleser, Marcus Rueckert and Henne Vogelsang for this fantastic and unique service for our openSUSE Community!
But time has come and resources went into other directions, so we need to find a new place for this service, which is meanwhile aggregating feeds from over 100 users of the openSUSE Community. As the legal analysis showed, the service itself is not critical - and (thanks to all those who write those fabulous blog entries) neither the content is.
As result, planet.opensuse.org is now running on a dedicated virtual host inside the openSUSE Cloud. Special thanks again to Marcus Rueckert and Henne Vogelsang who did the migration and Ciaran Farell who did the legal analysis.
Every former list member is added to the opensuse-admin group in this ticket system.
Note for new admin members: you have to log in at least once here, so we "see" you in the system and can assign you to the opensuse-admin group.
Also available in: Atom