action #61085

New Tumbleweed Snapshot (20191216) does not show up on QA

Added by dimstar 2 months ago. Updated 2 months ago.

Status:ResolvedStart date:17/12/2019
Priority:UrgentDue date:
Assignee:okurz% Done:

0%

Category:-
Target version:openQA Project - Current Sprint
Duration:

Description

Snapshot 1216 has been tagged for QA at 12/17 07:45 CET. Until now (11L30 CET), the snapshot does not show up on openQA

openqa_rsync.log contains a bunch of errors though:

error scheduling openSUSE-MicroOS-DVD-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: VERSION"error scheduling openSUSE-Kubic-DVD-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: VERSION"error scheduling openSUSE-Tumbleweed-NET-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-DVD-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-GNOME-Live-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-Rescue-CD-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"error scheduling openSUSE-Tumbleweed-XFCE-Live-x86_64-Snapshot20191216-Media.iso
400 Bad Request
"Error: missing parameters: DISTRI VERSION"rsync of tumbleweed done at Tue Dec 17 08:27:58 UTC 2019

History

#1 Updated by okurz 2 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz
  • Target version set to Current Sprint

Sounds like the deployment of 2019-12-17 on o3 webui caused this. Using https://github.com/okurz/scripts/blob/master/zypper-log-to-git-log and git log1 --no-merges $(zypper-log-to-git-log openQA) we can find

0924bf952 Provide git log diff from tests in investigation
c9c40894a Ignore selected entries from vars.json files for investigation
1f0c0d6ff t: Use more mojo Mojo::File instead of catfile/copy
fb71858bc Improve video player
62c5546ad t: Save execution time in 18-tests-details by disabling wait timeout when expecting 0 needles
b69d0a262 t: Add helper methods for selenium tests to disable/enable timeout to save time
0bac65271 Trigger tasks for limiting assets and results/logs hourly

#2 Updated by okurz 2 months ago

probably unrelated but still to fix, from /var/log/openqa_rsync.log-20191217.xz:

rsync of staging done at Fri Dec 13 10:50:03 UTC 2019
Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
Use of uninitialized value $val in numeric lt (<) at /usr/lib/perl5/vendor_perl/5.26.1/File/Rsync.pm line 548.
Use of uninitialized value in string eq at /opt/openqa-scripts/rsync_opensuse.pm line 424.
Use of uninitialized value in string eq at /opt/openqa-scripts/rsync_opensuse.pm line 424.
Use of uninitialized value in string eq at /opt/openqa-scripts/rsync_opensuse.pm line 424.
Use of uninitialized value in string eq at /opt/openqa-scripts/rsync_opensuse.pm line 424.
kernel-default-5.3.12-1.1.nosrc.rpm -> kernel-source-5.3.12-1.1.src.rpm
kernel-default-5.3.12-1.1.nosrc.rpm -> kernel-source-5.3.12-1.1.src.rpm
error scheduling Tumbleweed.x86_64-1.0-libvirt-Snapshot20191212.vagrant.libvirt.box
400 Bad Request
"Error: missing parameters: DISTRI VERSION FLAVOR ARCH"error scheduling Tumbleweed.x86_64-1.0-virtualbox-Snapshot20191212.vagrant.virtualbox.box
400 Bad Request
"Error: missing parameters: DISTRI VERSION FLAVOR ARCH"rsync of tumbleweed done at Fri Dec 13 11:17:29 UTC 2019
rsync of tumbleweed_ppc64 done at Fri Dec 13 11:17:31 UTC 2019

or after https://gitlab.suse.de/openqa/scripts/commit/30854a049a3e92aaad6e6709bade662473365ae7

rsync of staging done at Tue Dec 17 10:50:07 UTC 2019
Use of uninitialized value $_ in pattern match (m//) at /opt/openqa-scripts/rsync_opensuse.pm line 400.
Use of uninitialized value $_ in pattern match (m//) at /opt/openqa-scripts/rsync_opensuse.pm line 400.
settings for appliances/x86_64/kiwi-templates-JeOS:kvm-and-xen/openSUSE-Tumbleweed-JeOS.x86_64-15.1.0-kvm-and-xen-Snapshot20191216.qcow2 is neither hash nor sub!
Use of uninitialized value $_ in pattern match (m//) at /opt/openqa-scripts/rsync_opensuse.pm line 400.
Use of uninitialized value $_ in pattern match (m//) at /opt/openqa-scripts/rsync_opensuse.pm line 400.
Use of uninitialized value in sprintf at /opt/openqa-scripts/rsync_opensuse.pm line 487.
Use of uninitialized value in sprintf at /opt/openqa-scripts/rsync_opensuse.pm line 489.
Use of uninitialized value in sprintf at /opt/openqa-scripts/rsync_opensuse.pm line 492.
[…]
[17/12/2019 11:59:58] <okurz> but currently there is an rsync running for TW ppc64le
[17/12/2019 12:25:01] <okurz> I have rolled back the openQA webui installation. This seems to help with the asset management. https://openqa.opensuse.org/admin/assets is usable again and /space seems to recover. I would be surprised if this helps with triggering the TW snapshot but who knows :)
[17/12/2019 12:30:56] <okurz> for the TW snapshot syncing I can see that rsync.pl is just very busy right now copying the ppc snapshot but nothing wrong there yet
[17/12/2019 12:33:08] <okurz> I changed the log path of the cron job call from /var/log/openqa_rsync_staging.log to /var/log/openqa_rsync.log as that one is also used within the script itself. I consider the _staging file not that helpful anymore
[17/12/2019 12:33:15] <DimStar> okurz: probably not - in this case we'd have to clean up the 1216 stuff in /var/lib/openqa/factory, os rsync resync it
[17/12/2019 12:34:34] <okurz> well rsync.pl can be asked to force even if there are already isos present. that shouldn't be the problem. but it's currently just *blocked*. We can recheck after the ppc snapshot is done or should I try to abort that, try x86_64 first and restart with ppc64le manually?
[17/12/2019 12:35:46] <okurz> fvogt: scratch my last about trying a different git state. The warnings are already in way older versions, not the problem of today, red herring :)
[17/12/2019 12:37:13] <DimStar> okurz: nah, let's wait for ppc; there is not that much rush

#3 Updated by okurz 2 months ago

  • Status changed from In Progress to Resolved

so it looks like https://gitlab.suse.de/openqa/scripts/merge_requests/419/diffs#5732ff85e2f8643646047b834a854168560719b8_390_400 caused the problem. I thought it could not be because that should have been deployed already before yesterday but https://gitlab.suse.de/openqa/scripts/merge_requests/419 was merged 1 week ago but the deployment failed. good that no one cared! :D

So https://gitlab.suse.de/openqa/scripts/merge_requests/423 is a revert of the WSL changes causing problems. merged and deployed to o3. scripts are sane again on o3. https://gitlab.suse.de/openqa/scripts/merge_requests/422 would have been my proposed fixed but it would have fixed only parts of it. Seems there was the problem that e.g. DISTRI was never set to opensuse, etc.

https://gitlab.suse.de/openqa/scripts/merge_requests/424 to enable fatal warnings.

Also available in: Atom PDF