Project

General

Profile

action #59870

media are not synced anymore on o3

Added by okurz 7 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Start date:
2019-11-15
Due date:
% Done:

0%

Estimated time:
Duration:

Description

Observation

[15/11/2019 08:31:59] <DimStar> good morning all - seems the TW staging isos are no longer synced to QA :(
[15/11/2019 09:09:12] <okurz> good morning. I see some errors in /var/log/openqa_rsync.log: 400 Bad Request "Error: missing parameters: DISTRI VERSION FLAVOR ARCH"rsync of tumbleweed done at Thu Nov 14 06:55:32 UTC 2019 Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
[15/11/2019 09:14:30] <okurz> DimStar: there is an rsync process running since yesterday. AFAIK yesterday there was some work in the infrastructure that could have impacted us. I will try to fix this

processes on o3

geekote+  1816  0.0  0.0  13588  1400 ?        Ss   Nov14   0:00      \_ /bin/sh -c    /opt/openqa-scripts/openqa-iso-sync >> /var/log/openqa_rsync_staging.log 2>&1
geekote+  1819  0.0  0.0  44828  5296 ?        S    Nov14   0:00          \_ python /usr/bin/withlock -q /var/lib/openqa/.config/openqa/openqa-iso-sync.lock env LOCKED=1 /opt/openqa-scripts/openqa-iso-sync
geekote+  1824  0.0  0.0  13588  1448 ?        S    Nov14   0:00              \_ /bin/bash /opt/openqa-scripts/openqa-iso-sync
geekote+  1838  0.0  0.2 126348 46368 ?        S    Nov14   0:00                  \_ /usr/bin/perl -w /opt/openqa-scripts/rsync.pl --host localhost --repourl http://openqa.opensuse.org/assets/repo staging
geekote+  3641  0.0  0.0  27376  2880 ?        S    Nov14   0:05                      \_ rsync --checksum --verbose rsync://openqa@obs-back-stage.publish.opensuse.org/opensuse-internal/build/openSUSE:Factory:Staging:H/images/
geekote+  3652  0.0  0.0  27768  1916 ?        S    Nov14   0:03                          \_ rsync --checksum --verbose rsync://openqa@obs-back-stage.publish.opensuse.org/opensuse-internal/build/openSUSE:Factory:Staging:H/ima
r

History

#1 Updated by okurz 7 months ago

  • Status changed from In Progress to Feedback
  • Priority changed from Immediate to Normal

I terminated the stuck process. :G is running, maybe you are missing :H as that was the rsync process it was stuck on and I terminated. It seems we have a 2h timeout for the SLES rsync.pl call but not for openSUSE, probably because it was more stable the past 6 years or so ;)

https://gitlab.suse.de/openqa/scripts/blob/master/rsync.pl#L657 should get a parameter "timeout => 3600" as well.

https://gitlab.suse.de/openqa/scripts/merge_requests/416

#2 Updated by okurz 6 months ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF