tickets #111246
closed/srv/bs/bin/needed_push
100%
Description
I happened to glance at some of the output from this job (sent to admin-auto), from this morning - the mail is 644K, 13755 lines.
Surely that is ridiculous :-)
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '20.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '19.10': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '18.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '16.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Debian': No such file or directory
rm: cannot remove '10': No such file or directory
rm: cannot remove 'leap': No such file or directory
Further down:
rm: cannot remove 'home:p7k:zabbix:4.0_Fedora': No such file or directory
rm: cannot remove '30': No such file or directory
rm: cannot remove 'home:pagarcia:hackweek20:SLEMicro5.0-Uyuni-Client-Tools_MicroOS': No such file or directory
rm: cannot remove '5.0': No such file or directory
rm: cannot remove 'home:lahcim8_OpenSUSE': No such file or directory
rm: cannot remove 'Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'home:smplayerdev:tests_Fedora': No such file or directory
rm: cannot remove '22': No such file or directory
rm: cannot remove 'home:smplayerdev:tests_Fedora': No such file or directory
rm: cannot remove '23': No such file or directory
rm: cannot remove 'home:syrianidou_sofia_containers': No such file or directory
rm: cannot remove 'leap': No such file or directory
rm: cannot remove 'home:syrianidou_sofia_images': No such file or directory
rm: cannot remove 'leap': No such file or directory
rm: cannot remove 'home:Charadon_RHEL': No such file or directory
rm: cannot remove '8': No such file or directory
rm: cannot remove 'home:bjoern_beutel:private_Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'home:bjoern_beutel:private_Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'Update': No such file or directory
Updated by pjessen over 2 years ago
- Private changed from Yes to No
A closer look at the output suggests it is about file or directory names containing blanks ....
E.g. in /srv/bs/arrived :
science:SimSpark:SimSpark-SPL-rUNSWift_Debian 10
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 16.04
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 18.04
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 19.10
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 20.04
Two issues:
/srv/bs/bin/needed_push::11 - the for loop (default field sep is blank).
/srv/bs/bin/needed_push::16 - the rm.
Updated by pjessen over 2 years ago
The for loop is easily fixed with
(find . -type f ! -name home\*
find . -type f -name home\*) |\
while read i
do
However, that one doesn't deal with trailing blanks ...
This is the solution:
while IFS= read -r -d $'\0' f
do
do-the-stuff ...
done < <(find . -type f ! -name home\* -print0; find . -type f -name home\* -print0)
Updated by pjessen over 2 years ago
- Status changed from New to In Progress
- Assignee set to pjessen
- % Done changed from 0 to 50
It appears that the contents of the files arrived/whatever-to-push are null-terminated ...
Updated by pjessen over 2 years ago
Note to self: I think this needs fixing in /srv/bs/bin/run_push too.
Updated by pjessen over 2 years ago
Still a couple of issues:
rm: cannot remove 'home:/rawtherapee/Debian_Next_ga ': No such file or directory
rm: cannot remove 'home:/cchengMW:/tine20drive:/2.6.1/Ubuntu_18.04 ': No such file or directory
rm: cannot remove 'home:/cchengMW:/tine20drive:/2.6.1/Ubuntu_18.10 ': No such file or directory
I think those may be left overs from a previous era, I'll just delete them manually.
Updated by pjessen over 2 years ago
- Due date changed from 2022-05-20 to 2022-08-08
Updated by pjessen over 2 years ago
- Related to tickets #113632: How to request for mirror rsync adjustments? added
Updated by pjessen over 2 years ago
I am continuing the discussion here - judging by /srv/bs/arrived on pontifex, the repo push is way behind. /srv/bs/arrived holds 23019 repos due to be pushed, some dating back to 18 May. We are pushing repos to 9 mirrors, but only one at a time.
The run_push scripts goes through the nine servers in /srv/bs/servers, then picks one(!) repo from /srv/bs/arrived in order of oldest, with non-home repos before home-repos. Looking only at July 2022, we are pushing an average of 439 repos to one mirror, every hour. That's pathetic.
I think we have to:
a) do concurrent syncs to all push mirrors, and
b) do batches of repos, not just one per go.
Sounds like a fun project for my summer hols :-)
Updated by pjessen over 2 years ago
- % Done changed from 50 to 30
pjessen wrote:
I think we have to:
a) do concurrent syncs to all push mirrors, and
b) do batches of repos, not just one per go.
Sounds like a fun project for my summer hols :-)
Note to self - I have slowly begun to develop some ideas, currently in my homedir on pontifex.
Updated by pjessen over 2 years ago
In /srv/bs/bin/needed_push, I have enabled the removal of files from arrival/. I think that might help with the "backlog".
Updated by pjessen over 2 years ago
- % Done changed from 30 to 50
pjessen wrote:
Note to self - I have slowly begun to develop some ideas, currently in my homedir on pontifex.
I put my new scheme into production last week, 28 August. At first it did not seem to help much, but it is now improving. It looks like we're catching up.
Updated by pjessen over 1 year ago
- Status changed from In Progress to Resolved
- % Done changed from 50 to 100
Did we catch up yet?
To be honest, I don't know - some other scheme was put in place.