tickets #115640
openprovo-mirror.o.o/rsync.o.o - review of rsync process
0%
Description
I was looking at other tickets involving the rsync on provo-mirror when I noticed the rsync of repos seemed to had gotten stuck.
With strace I see it repeating a select() with a 60 second timeout, endlessly.
The main loop /home/mirror/bin/loop_repositories.sh was started Thu Aug 11 13:14:47 2022, followed by the actual rsync at Fri Aug 12 20:47:30 2022. The log file has not been written to since 2022-08-13 00:00:08. Not being used to handling 'screen', I managed to restart it, but at the very least, the rsync command needs a timeout setting.
Updated by pjessen over 1 year ago
- Subject changed from provo-mirror - review of rsync process to provo-mirror.o.o/rsync.o.o - review of rsync process
- Category set to Servers hosted in Provo
- Private changed from Yes to No
Lars, I guess you might be reading this too :-)
TBH, I'm not really very keen on the cscreen setup on provo-mirror and rsync. I am missing regular logfiles (with rotation), and some sort of email alert when something's wrong. I'm going to use this ticket to discuss/record changes I'm making or would like to make.
- 202208xx both : have added a loop_debug script (see #115091)
- 20220823 provo: have added logrotate for rsyncd-push.log (daily, 90 generations)
- 20220823 both : have enabled ip4/6 in all scripts
- 20220823 both : have added --stop-after=60 in all scripts (might need reviewing)
- 20220823 both : have amended loop_repositories to prioritise none-home repos
Updated by pjessen over 1 year ago
Thoughts -
- it would be good to run just one rsync at a time. Running multiple will only slow them all down.
- some rsyncs are long running and would block others (which might be more important).
- ports/ takes up 2.5Tb
- debug/ takes up 1.5Tb
- repositories/ takes up 25Tb (nonhome 12Tb, home 13Tb)
- for those big ones above, I wonder if it might be useful to run them in chunks, e.g. 1 hour at a time, then take a break. I tried this with
--stop-after, but it didn't quite work, maybe due to--delay-updates
Updated by pjessen over 1 year ago
- Related to tickets #100506: rsync.o.o setup added
Updated by pjessen over 1 year ago
On provo-mirror, the repositories/ sync has been running for almost 48 hours:
# ps -p 12178 -o lstart
Fri Aug 26 10:30:14 2022
# date
Sun Aug 28 09:48:06 UTC 2022