tickets #111246
closed
Added by pjessen over 2 years ago.
Updated over 1 year ago.
Description
I happened to glance at some of the output from this job (sent to admin-auto), from this morning - the mail is 644K, 13755 lines.
Surely that is ridiculous :-)
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '20.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '19.10': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '18.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '16.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Debian': No such file or directory
rm: cannot remove '10': No such file or directory
rm: cannot remove 'leap': No such file or directory
Further down:
rm: cannot remove 'home:p7k:zabbix:4.0_Fedora': No such file or directory
rm: cannot remove '30': No such file or directory
rm: cannot remove 'home:pagarcia:hackweek20:SLEMicro5.0-Uyuni-Client-Tools_MicroOS': No such file or directory
rm: cannot remove '5.0': No such file or directory
rm: cannot remove 'home:lahcim8_OpenSUSE': No such file or directory
rm: cannot remove 'Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'home:smplayerdev:tests_Fedora': No such file or directory
rm: cannot remove '22': No such file or directory
rm: cannot remove 'home:smplayerdev:tests_Fedora': No such file or directory
rm: cannot remove '23': No such file or directory
rm: cannot remove 'home:syrianidou_sofia_containers': No such file or directory
rm: cannot remove 'leap': No such file or directory
rm: cannot remove 'home:syrianidou_sofia_images': No such file or directory
rm: cannot remove 'leap': No such file or directory
rm: cannot remove 'home:Charadon_RHEL': No such file or directory
rm: cannot remove '8': No such file or directory
rm: cannot remove 'home:bjoern_beutel:private_Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'home:bjoern_beutel:private_Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'Update': No such file or directory
- Private changed from Yes to No
A closer look at the output suggests it is about file or directory names containing blanks ....
E.g. in /srv/bs/arrived :
science:SimSpark:SimSpark-SPL-rUNSWift_Debian 10
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 16.04
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 18.04
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 19.10
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 20.04
Two issues:
/srv/bs/bin/needed_push::11 - the for loop (default field sep is blank).
/srv/bs/bin/needed_push::16 - the rm.
The for loop is easily fixed with
(find . -type f ! -name home\*
find . -type f -name home\*) |\
while read i
do
However, that one doesn't deal with trailing blanks ...
This is the solution:
while IFS= read -r -d $'\0' f
do
do-the-stuff ...
done < <(find . -type f ! -name home\* -print0; find . -type f -name home\* -print0)
- Status changed from New to In Progress
- Assignee set to pjessen
- % Done changed from 0 to 50
It appears that the contents of the files arrived/whatever-to-push are null-terminated ...
- Due date set to 2022-05-20
Note to self: I think this needs fixing in /srv/bs/bin/run_push too.
Still a couple of issues:
rm: cannot remove 'home:/rawtherapee/Debian_Next_ga ': No such file or directory
rm: cannot remove 'home:/cchengMW:/tine20drive:/2.6.1/Ubuntu_18.04 ': No such file or directory
rm: cannot remove 'home:/cchengMW:/tine20drive:/2.6.1/Ubuntu_18.10 ': No such file or directory
I think those may be left overs from a previous era, I'll just delete them manually.
- Due date changed from 2022-05-20 to 2022-08-08
- Related to tickets #113632: How to request for mirror rsync adjustments? added
I am continuing the discussion here - judging by /srv/bs/arrived on pontifex, the repo push is way behind. /srv/bs/arrived holds 23019 repos due to be pushed, some dating back to 18 May. We are pushing repos to 9 mirrors, but only one at a time.
The run_push scripts goes through the nine servers in /srv/bs/servers, then picks one(!) repo from /srv/bs/arrived in order of oldest, with non-home repos before home-repos. Looking only at July 2022, we are pushing an average of 439 repos to one mirror, every hour. That's pathetic.
I think we have to:
a) do concurrent syncs to all push mirrors, and
b) do batches of repos, not just one per go.
Sounds like a fun project for my summer hols :-)
- % Done changed from 50 to 30
pjessen wrote:
I think we have to:
a) do concurrent syncs to all push mirrors, and
b) do batches of repos, not just one per go.
Sounds like a fun project for my summer hols :-)
Note to self - I have slowly begun to develop some ideas, currently in my homedir on pontifex.
In /srv/bs/bin/needed_push, I have enabled the removal of files from arrival/. I think that might help with the "backlog".
- % Done changed from 30 to 50
pjessen wrote:
Note to self - I have slowly begun to develop some ideas, currently in my homedir on pontifex.
I put my new scheme into production last week, 28 August. At first it did not seem to help much, but it is now improving. It looks like we're catching up.
- Status changed from In Progress to Resolved
- % Done changed from 50 to 100
Did we catch up yet?
To be honest, I don't know - some other scheme was put in place.
Also available in: Atom
PDF