probably unrelated but still to fix, from /var/log/openqa_rsync.log-20191217.xz:
rsync of staging done at Fri Dec 13 10:50:03 UTC 2019
Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
Use of uninitialized value in string eq at /opt/openqa-scripts/rsync_opensuse.pm line 424.
Use of uninitialized value in string eq at /opt/openqa-scripts/rsync_opensuse.pm line 424.
Use of uninitialized value in string eq at /opt/openqa-scripts/rsync_opensuse.pm line 424.
Use of uninitialized value in string eq at /opt/openqa-scripts/rsync_opensuse.pm line 424.
kernel-default-5.3.12-1.1.nosrc.rpm -> kernel-source-5.3.12-1.1.src.rpm
kernel-default-5.3.12-1.1.nosrc.rpm -> kernel-source-5.3.12-1.1.src.rpm
error scheduling Tumbleweed.x86_64-1.0-libvirt-Snapshot20191212.vagrant.libvirt.box
400 Bad Request
"Error: missing parameters: DISTRI VERSION FLAVOR ARCH"error scheduling Tumbleweed.x86_64-1.0-virtualbox-Snapshot20191212.vagrant.virtualbox.box
400 Bad Request
"Error: missing parameters: DISTRI VERSION FLAVOR ARCH"rsync of tumbleweed done at Fri Dec 13 11:17:29 UTC 2019
rsync of tumbleweed_ppc64 done at Fri Dec 13 11:17:31 UTC 2019
or after https://gitlab.suse.de/openqa/scripts/commit/30854a049a3e92aaad6e6709bade662473365ae7
rsync of staging done at Tue Dec 17 10:50:07 UTC 2019
Use of uninitialized value $_ in pattern match (m//) at /opt/openqa-scripts/rsync_opensuse.pm line 400.
Use of uninitialized value $_ in pattern match (m//) at /opt/openqa-scripts/rsync_opensuse.pm line 400.
settings for appliances/x86_64/kiwi-templates-JeOS:kvm-and-xen/openSUSE-Tumbleweed-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot20191216.qcow2 is neither hash nor sub!
Use of uninitialized value $_ in pattern match (m//) at /opt/openqa-scripts/rsync_opensuse.pm line 400.
Use of uninitialized value $_ in pattern match (m//) at /opt/openqa-scripts/rsync_opensuse.pm line 400.
Use of uninitialized value in sprintf at /opt/openqa-scripts/rsync_opensuse.pm line 487.
Use of uninitialized value in sprintf at /opt/openqa-scripts/rsync_opensuse.pm line 489.
Use of uninitialized value in sprintf at /opt/openqa-scripts/rsync_opensuse.pm line 492.
[…]
[17/12/2019 11:59:58] <okurz> but currently there is an rsync running for TW ppc64le
[17/12/2019 12:25:01] <okurz> I have rolled back the openQA webui installation. This seems to help with the asset management. https://openqa.opensuse.org/admin/assets is usable again and /space seems to recover. I would be surprised if this helps with triggering the TW snapshot but who knows :)
[17/12/2019 12:30:56] <okurz> for the TW snapshot syncing I can see that rsync.pl is just very busy right now copying the ppc snapshot but nothing wrong there yet
[17/12/2019 12:33:08] <okurz> I changed the log path of the cron job call from /var/log/openqa_rsync_staging.log to /var/log/openqa_rsync.log as that one is also used within the script itself. I consider the _staging file not that helpful anymore
[17/12/2019 12:33:15] <DimStar> okurz: probably not - in this case we'd have to clean up the 1216 stuff in /var/lib/openqa/factory, os rsync resync it
[17/12/2019 12:34:34] <okurz> well rsync.pl can be asked to force even if there are already isos present. that shouldn't be the problem. but it's currently just *blocked*. We can recheck after the ppc snapshot is done or should I try to abort that, try x86_64 first and restart with ppc64le manually?
[17/12/2019 12:35:46] <okurz> fvogt: scratch my last about trying a different git state. The warnings are already in way older versions, not the problem of today, red herring :)
[17/12/2019 12:37:13] <DimStar> okurz: nah, let's wait for ppc; there is not that much rush