Project

General

Profile

action #108896

openQA Tests - action #107062: Multiple failures due to network issues

[ppc64le] auto_review:"(?s)Size of.*differs, expected.*but downloaded.*Download.*failed: 521 Connect timeout":retry

Added by okurz 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2022-03-24
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://openqa.suse.de/tests/8378592 is incomplete with

[2022-03-24T22:00:53.638080+01:00] [info] [pid:59369] Download of sle-12-SP4-ppc64le-sap-nw-noscc.qcow2 processed:
[info] [#30431]
Cache size of "/var/lib/openqa/cache" is 44 GiB, with limit 50 GiB
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Size of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" differs, expected 10 GiB but downloaded 6.3 GiB
[info] [#30431]
Download error 598, waiting 5 seconds for next try (4 remaining)
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Download of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" failed: 521 Connect timeout
[info] [#30431]
Download error 521, waiting 5 seconds for next try (3 remaining)
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Download of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" failed: 521 Connect timeout
[info] [#30431]
Download error 521, waiting 5 seconds for next try (2 remaining)
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Size of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" differs, expected 10 GiB but downloaded 2.6 GiB
[info] [#30431]
Download error 598, waiting 5 seconds for next try (1 remaining)
[info] [#30431]
Downloading "sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" from "http://openqa.suse.de/tests/8378592/asset/hdd/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2"
[info] [#30431]
Download of "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" failed: 521 Connect timeout
[info] [#30431]
Purging "/var/lib/openqa/cache/openqa.suse.de/sle-12-SP4-ppc64le-sap-nw-noscc.qcow2" because of too many download errors

so severe download problems. Also I saw on https://stats.openqa-monitor.qa.suse.de/d/WDQA-Power8-4-kvm/worker-dashboard-qa-power8-4-kvm?viewPanel=65109&orgId=1&refresh=1m&from=now-2d&to=now that the download speed is really low. I also paused the alert as it was already triggering yesterday.

Steps to reproduce

Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
openqa-query-for-job-label poo#108896

Rollback steps

  • Unpause download rate alert for qa-power8-4

Related issues

Related to openQA Infrastructure - action #108845: Network performance problems, DNS, DHCP, within SUSE QA network auto_review:"(Error connecting to VNC server.*qa.suse.*Connection timed out|ipmitool.*qa.suse.*Unable to establish)":retry but also other symptoms size:MResolved2022-03-242022-04-15

History

#1 Updated by okurz 3 months ago

  • Related to action #108845: Network performance problems, DNS, DHCP, within SUSE QA network auto_review:"(Error connecting to VNC server.*qa.suse.*Connection timed out|ipmitool.*qa.suse.*Unable to establish)":retry but also other symptoms size:M added

#2 Updated by okurz 3 months ago

  • Description updated (diff)

#3 Updated by okurz 3 months ago

  • Parent task set to #107062

I called export host=openqa.suse.de; bash -ex ./openqa-monitor-incompletes | bash -e ./openqa-label-known-issues and found 63 unknown issues, most of them about "Error connecting to VNC server.*s390", noted in another ticket as well.

#4 Updated by okurz 3 months ago

  • Status changed from New to Blocked
  • Assignee set to okurz

#5 Updated by okurz 3 months ago

  • Status changed from Blocked to Resolved

"QA-Power8-4-kvm: Download rate alert" is green. I unpaused the alert again.

openqa-query-for-job-label poo#108896 returns as most recent results an entry from 2022-04-02 so we should be good.

Also available in: Atom PDF