action #60959

Seems we have many incompletes since 2019-12-12 on o3 , failing just in the middle of test modules with no logs uploaded

Added by okurz 3 months ago. Updated 3 months ago.

Status:ResolvedStart date:12/12/2019
Priority:HighDue date:
Assignee:okurz% Done:

0%

Category:-
Target version:openQA Project - Done
Duration:

Description

Observation

https://openqa.opensuse.org/tests/?&resultfilter=Incomplete shows many jobs incomplete failing just in the middle of test modules with no logs uploaded.

package diff on w1.o.o:

# grep '2019-12-1[12]' /var/log/zypp/history
2019-12-11 00:16:10|command|root@openqaworker1|'zypper' '-R' '/tmp/tmp.DOTG4zdY9d' 'up' '-y' '--auto-agree-with-product-licenses'|
2019-12-11 00:16:18|install|os-autoinst|4.6.1576005857.163a29fa-lp151.295.1|x86_64||devel_openQA|55801d0ba861fc7f1280583ca89ff3ddec051b2c05870c15d26a475ee0273d78|
# 2019-12-11 00:16:23 os-autoinst-openvswitch-4.6.1576005857.163a29fa-lp151.295.1.x86_64.rpm installed ok
2019-12-11 00:16:23|install|os-autoinst-openvswitch|4.6.1576005857.163a29fa-lp151.295.1|x86_64||devel_openQA|bb579718d155cef724b33c8149f4d4585b591ffc05eae7c4ff74014c3784f2e4|
2019-12-11 00:16:27|install|openQA-common|4.6.1575990828.76dde5b6f-lp151.2085.1|noarch||devel_openQA|038cf1f0500e936287cd051a0bc6c171d39e90d23161d7e2a240cf99e8032a8e|
2019-12-11 00:16:28|install|openQA-client|4.6.1575990828.76dde5b6f-lp151.2085.1|noarch||devel_openQA|9acc3128066fd82ffff4ae1c671fe4085c1b2acecc28b4396b1131ad9ac0f61d|
# 2019-12-11 00:16:30 openQA-worker-4.6.1575990828.76dde5b6f-lp151.2085.1.noarch.rpm installed ok
2019-12-11 00:16:30|install|openQA-worker|4.6.1575990828.76dde5b6f-lp151.2085.1|noarch||devel_openQA|73d579a42cf8c4ddc724e5ee1a039527e558c1e8c3e6f06a9ce9e4690a33f762|
2019-12-12 01:32:08|command|root@openqaworker1|'zypper' '-R' '/tmp/tmp.JddE3uTMEi' 'up' '-y' '--auto-agree-with-product-licenses'|
2019-12-12 01:32:17|install|perl-Perl-Tidy|20191203-lp151.2.1|noarch||devel_openQA_Leap_15.1|94a3fd23ae9256e0eb402937d1bcc4db59ff6412fd47f4155ca48737a41ffb0d|
2019-12-12 01:32:28|install|libopencv3_3|3.3.1-lp151.6.3.1|x86_64||repo-update|d2180bc0ba5817d462e00f01933ec2f79029b41fad1934b68edcef2c737ecf9b|
# 2019-12-12 01:32:32 permissions-20181116-lp151.4.9.1.x86_64.rpm installed ok
2019-12-12 01:32:32|install|permissions|20181116-lp151.4.9.1|x86_64||repo-update|6f3b4ebe5be93d8d053202b548c7e41af6ddbb09c9538d89044a418042e6686f|
2019-12-12 01:32:33|install|os-autoinst|4.6.1576050328.08c055a7-lp151.297.1|x86_64||devel_openQA|0e09ff795531fa1bf54629afc2a1d953b24decf7460f0cc7a879edfd28c2f546|
# 2019-12-12 01:32:36 os-autoinst-openvswitch-4.6.1576050328.08c055a7-lp151.297.1.x86_64.rpm installed ok
2019-12-12 01:32:36|install|os-autoinst-openvswitch|4.6.1576050328.08c055a7-lp151.297.1|x86_64||devel_openQA|f4b4a4a9c7fcf7ae6101e3c4ed6b711eb88deac4f11de12d64179fc9864a10b8|
2019-12-12 01:32:38|install|openQA-common|4.6.1576103349.a77e8a6af-lp151.2092.1|noarch||devel_openQA|9ff50f7531da564baf7603da5be5c86a3ddcedc9a4af3e107b383166564d0716|
2019-12-12 01:32:39|install|openQA-client|4.6.1576103349.a77e8a6af-lp151.2092.1|noarch||devel_openQA|9e750ebaffde7fc15537b512a6d722f7b5117f9d1a14cbc42e013b67642c9a06|
# 2019-12-12 01:32:41 openQA-worker-4.6.1576103349.a77e8a6af-lp151.2092.1.noarch.rpm installed ok
2019-12-12 01:32:41|install|openQA-worker|4.6.1576103349.a77e8a6af-lp151.2092.1|noarch||devel_openQA|7bc03fb54ad9cd8e534f9f5d496ef267e4caf75c77844d09d32d06e6ece82046|

and for webUI:

$ git log1 --no-merges $(zypper-log-to-git-log openQA)
3d8467040 Missing assets should not result in a connection timeout
211068d66 Fix regex and error message for prio
984d3fb5e Parser::Format::IPA: Shorten test names
812508b9a Avoid downloads from GitHub within 14-grutasks.t
f0d70f1b2 Improve asset download
690cbc743 Use human readable size values in cache service logs

History

#1 Updated by okurz 3 months ago

  • Status changed from In Progress to Feedback
  • Priority changed from Urgent to High

maybe it's because of the weekly IT maintenance work. I observed other network problems as well. I retriggered all recent incomplete jobs and will monitor them.

It seems to be better now, e.g.
https://openqa.opensuse.org/tests/1110877 is just fine, others also look good.

#2 Updated by okurz 3 months ago

  • Project changed from openQA Project to openQA Infrastructure
  • Category deleted (Concrete Bugs)
  • Status changed from Feedback to Resolved
  • Target version changed from Current Sprint to Done

no further problems detected. I will accept my initial hypothesis about the maintenance window.

Also available in: Atom PDF