action #108896
closedopenQA Tests - action #107062: Multiple failures due to network issues
[ppc64le] auto_review:"(?s)Size of.*differs, expected.*but downloaded.*Download.*failed: 521 Connect timeout":retry
0%
Description
Observation¶
https://openqa.suse.de/tests/8378592 is incomplete with
[2022-03-24T22:00:53.638080+01:00] [info] [pid:59369] Download of sle-12-SP4-ppc64le-sap-nw-noscc.qcow2 processed:
[info] [#30431]
Cache size of "/var/lib/openqa/cache" is 44 GiB, with limit 50 GiB
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Size of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" differs, expected 10 GiB but downloaded 6.3 GiB
[info] [#30431]
Download error 598, waiting 5 seconds for next try (4 remaining)
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Download of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" failed: 521 Connect timeout
[info] [#30431]
Download error 521, waiting 5 seconds for next try (3 remaining)
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Download of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" failed: 521 Connect timeout
[info] [#30431]
Download error 521, waiting 5 seconds for next try (2 remaining)
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Size of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" differs, expected 10 GiB but downloaded 2.6 GiB
[info] [#30431]
Download error 598, waiting 5 seconds for next try (1 remaining)
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Download of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" failed: 521 Connect timeout
[info] [#30431]
Purging "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" because of too many download errors
so severe download problems. Also I saw on https://stats.openqa-monitor.qa.suse.de/d/WDQA-Power8-4-kvm/worker-dashboard-qa-power8-4-kvm?viewPanel=65109&orgId=1&refresh=1m&from=now-2d&to=now that the download speed is really low. I also paused the alert as it was already triggering yesterday.
Steps to reproduce¶
Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
openqa-query-for-job-label poo#108896
Rollback steps¶
- Unpause download rate alert for qa-power8-4
Updated by okurz over 2 years ago
- Related to action #108845: Network performance problems, DNS, DHCP, within SUSE QA network auto_review:"(Error connecting to VNC server.*qa.suse.*Connection timed out|ipmitool.*qa.suse.*Unable to establish)":retry but also other symptoms size:M added
Updated by okurz over 2 years ago
- Parent task set to #107062
I called export host=openqa.suse.de; bash -ex ./openqa-monitor-incompletes | bash -e ./openqa-label-known-issues and found 63 unknown issues, most of them about "Error connecting to VNC server.*s390", noted in another ticket as well.
Updated by okurz over 2 years ago
- Status changed from New to Blocked
- Assignee set to okurz
Updated by okurz over 2 years ago
- Status changed from Blocked to Resolved
"QA-Power8-4-kvm: Download rate alert" is green. I unpaused the alert again.
openqa-query-for-job-label poo#108896 returns as most recent results an entry from 2022-04-02 so we should be good.