Project

General

Profile

Actions

tickets #111246

closed

/srv/bs/bin/needed_push

Added by pjessen almost 2 years ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Mirrors
Target version:
-
Start date:
2022-05-18
Due date:
2022-08-08
% Done:

100%

Estimated time:

Description

I happened to glance at some of the output from this job (sent to admin-auto), from this morning - the mail is 644K, 13755 lines.
Surely that is ridiculous :-)

rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '20.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '19.10': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '18.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu': No such file or directory
rm: cannot remove '16.04': No such file or directory
rm: cannot remove 'science:SimSpark:SimSpark-SPL-rUNSWift_Debian': No such file or directory
rm: cannot remove '10': No such file or directory
rm: cannot remove 'leap': No such file or directory

Further down:

rm: cannot remove 'home:p7k:zabbix:4.0_Fedora': No such file or directory
rm: cannot remove '30': No such file or directory
rm: cannot remove 'home:pagarcia:hackweek20:SLEMicro5.0-Uyuni-Client-Tools_MicroOS': No such file or directory
rm: cannot remove '5.0': No such file or directory
rm: cannot remove 'home:lahcim8_OpenSUSE': No such file or directory
rm: cannot remove 'Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'home:smplayerdev:tests_Fedora': No such file or directory
rm: cannot remove '22': No such file or directory
rm: cannot remove 'home:smplayerdev:tests_Fedora': No such file or directory
rm: cannot remove '23': No such file or directory
rm: cannot remove 'home:syrianidou_sofia_containers': No such file or directory
rm: cannot remove 'leap': No such file or directory
rm: cannot remove 'home:syrianidou_sofia_images': No such file or directory
rm: cannot remove 'leap': No such file or directory
rm: cannot remove 'home:Charadon_RHEL': No such file or directory
rm: cannot remove '8': No such file or directory
rm: cannot remove 'home:bjoern_beutel:private_Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'home:bjoern_beutel:private_Leap': No such file or directory
rm: cannot remove '15.3': No such file or directory
rm: cannot remove 'Update': No such file or directory

Related issues 1 (1 open0 closed)

Related to openSUSE admin - tickets #113632: How to request for mirror rsync adjustments?Newpjessen2022-07-14

Actions
Actions #1

Updated by pjessen almost 2 years ago

  • Private changed from Yes to No

A closer look at the output suggests it is about file or directory names containing blanks ....

E.g. in /srv/bs/arrived :

science:SimSpark:SimSpark-SPL-rUNSWift_Debian 10     
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 16.04
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 18.04
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 19.10
science:SimSpark:SimSpark-SPL-rUNSWift_Ubuntu 20.04

Two issues:
/srv/bs/bin/needed_push::11 - the for loop (default field sep is blank).
/srv/bs/bin/needed_push::16 - the rm.

Actions #2

Updated by pjessen almost 2 years ago

The for loop is easily fixed with

        (find . -type f ! -name home\*
        find . -type f -name home\*) |\
        while read i
        do

However, that one doesn't deal with trailing blanks ...

This is the solution:

        while IFS= read -r -d $'\0' f
        do
            do-the-stuff ...
        done < <(find . -type f ! -name home\* -print0; find . -type f -name home\* -print0)
Actions #3

Updated by pjessen almost 2 years ago

  • Status changed from New to In Progress
  • Assignee set to pjessen
  • % Done changed from 0 to 50

It appears that the contents of the files arrived/whatever-to-push are null-terminated ...

Actions #4

Updated by pjessen almost 2 years ago

  • Due date set to 2022-05-20
Actions #5

Updated by pjessen almost 2 years ago

Note to self: I think this needs fixing in /srv/bs/bin/run_push too.

Actions #6

Updated by pjessen almost 2 years ago

Still a couple of issues:

rm: cannot remove 'home:/rawtherapee/Debian_Next_ga ': No such file or directory
rm: cannot remove 'home:/cchengMW:/tine20drive:/2.6.1/Ubuntu_18.04 ': No such file or directory
rm: cannot remove 'home:/cchengMW:/tine20drive:/2.6.1/Ubuntu_18.10 ': No such file or directory

I think those may be left overs from a previous era, I'll just delete them manually.

Actions #7

Updated by lrupp almost 2 years ago

  • Category set to Mirrors
Actions #8

Updated by pjessen over 1 year ago

  • Due date changed from 2022-05-20 to 2022-08-08
Actions #9

Updated by pjessen over 1 year ago

  • Related to tickets #113632: How to request for mirror rsync adjustments? added
Actions #10

Updated by pjessen over 1 year ago

I am continuing the discussion here - judging by /srv/bs/arrived on pontifex, the repo push is way behind. /srv/bs/arrived holds 23019 repos due to be pushed, some dating back to 18 May. We are pushing repos to 9 mirrors, but only one at a time.
The run_push scripts goes through the nine servers in /srv/bs/servers, then picks one(!) repo from /srv/bs/arrived in order of oldest, with non-home repos before home-repos. Looking only at July 2022, we are pushing an average of 439 repos to one mirror, every hour. That's pathetic.
I think we have to:
a) do concurrent syncs to all push mirrors, and
b) do batches of repos, not just one per go.

Sounds like a fun project for my summer hols :-)

Actions #11

Updated by pjessen over 1 year ago

  • % Done changed from 50 to 30

pjessen wrote:

I think we have to:
a) do concurrent syncs to all push mirrors, and
b) do batches of repos, not just one per go.

Sounds like a fun project for my summer hols :-)

Note to self - I have slowly begun to develop some ideas, currently in my homedir on pontifex.

Actions #12

Updated by pjessen over 1 year ago

In /srv/bs/bin/needed_push, I have enabled the removal of files from arrival/. I think that might help with the "backlog".

Actions #13

Updated by pjessen over 1 year ago

  • % Done changed from 30 to 50

pjessen wrote:

Note to self - I have slowly begun to develop some ideas, currently in my homedir on pontifex.

I put my new scheme into production last week, 28 August. At first it did not seem to help much, but it is now improving. It looks like we're catching up.

Actions #14

Updated by crameleon 10 months ago

Did we catch up yet?

Actions #15

Updated by pjessen 10 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Did we catch up yet?

To be honest, I don't know - some other scheme was put in place.

Actions

Also available in: Atom PDF