Project

General

Profile

Actions
Copy link

action #115340

closed

[qem] test fails in add_update_test_repo[and zypper_lifecycle], seems performance issue when accessing maintenance repositories

Added by rfan1 over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
andriinikitin
Category:
Bugs in existing tests
Target version:
-
Start date:
2022-08-16
Due date:
% Done:

100%

Estimated time:
Difficulty:
Tags:
bugbusters

Description

Observation

openQA test in scenario sle-15-SP4-Server-DVD-Updates-x86_64-mau-webserver+15@64bit fails in
add_update_test_repo

Test suite description

Testsuite maintained at https://gitlab.suse.de/qa-maintenance/qam-openqa-yml. SLE15+ version of mau-webserver

Reproducible

Fails since (at least) Build 20220815-1

Expected result

Last good: 20220814-1 (or more recent)

Further details

Always latest result in this scenario: latest

Related issues 1 (0 open — 1 closed)

Related to openQA Tests - action #115406: [qe-core] needle fails on add_update_test_repoRejected2022-08-17

Actions
Actions
Copy link
 #1

Updated by rfan1 over 1 year ago

Lots of Core Maintenance Updates tests failed at add_update_test_repo and zypper_lifecycle. seems we are hitting some performance issues when accessing maintenance repositories.

Actions
Copy link
 #2

Updated by apappas over 1 year ago

  • Tags set to bugbusters

@rfan1 when you make tickets like that please add the bugbusters tag.

Actions
Copy link
 #3

Updated by rfan1 over 1 year ago

As Michael Andres's kindly reply, there might be some problem with download.suse.de

Even small requests resolved by download.suse.de itself take time:
+++++++++++++++++++++++++++
+++ 04:42:49 http://download.suse.de/ibs/SUSE:/Maintenance:/25413/SUSE_Updates_SLE-Module-Containers_15-SP4_s390x/repodata/repomd.xml.key
++ 04:42:54 HTTP response: 200
The slow connection to download.suse.de is probably nothing zypper can influence

Actions
Copy link
 #4

Updated by rfan1 over 1 year ago

Add one more commet:

I checked one of the failed case, and I can see lots of error messages in serial console,
https://openqa.suse.de/tests/9345669/logfile?filename=serial0.txt

2022-08-16 18:59:20 <5> install(17374) [zypp] Exception.cc(log):166 MediaCurl.cc(evaluateCurlCode):1126 THROW:    File '/repo/repoindex.xml' not found on medium 'http://download.suse.de/ibs/SUSE:/Maintenance:/24571/SUSE_Updates_SLE-SERVER_12-SP5_s390x/'
2022-08-16 18:59:20 <3> install(17374) [zypp] MediaCurl.cc(doGetFileCopyFile):1575 curl error: 22: The requested URL returned error: 404 Not Found, temp file size 0 bytes.
2022-08-16 18:59:20 <5> install(17374) [zypp] Exception.cc(log):166 MediaCurl.cc(evaluateCurlCode):1126 THROW:    File './media.1/products' not found on medium 'http://download.suse.de/ibs/SUSE:/Maintenance:/24571/SUSE_Updates_SLE-SERVER_12-SP5_s390x/'
2022-08-16 18:59:20 <5> install(17374) [zypp] Exception.cc(log):166 MediaCurl.cc(doGetFileCopyFile):1591 RETHROW:  File './media.1/products' not found on medium 'http://download.suse.de/ibs/SUSE:/Maintenance:/24571/SUSE_Updates_SLE-SERVER_12-SP5_s390x/'
2022-08-16 18:59:20 <5> install(17374) [zypp] Exception.cc(log):166 MediaMultiCurl.cc(doGetFileCopy):1363 RETHROW:  File './media.1/products' not found on medium 'http://download.suse.de/ibs/SUSE:/Maintenance:/24571/SUSE_Updates_SLE-SERVER_12-SP5_s390x/'
2022-08-16 18:59:20 <5> install(17374) [zypp] Exception.cc(log):166 MediaCurl.cc(getFileCopy):1023 RETHROW:  File './media.1/products' not found on medium 'http://download.suse.de/ibs/SUSE:/Maintenance:/24571/SUSE_Updates_SLE-SERVER_12-SP5_s390x/'
2022-08-16 18:59:20 <5> install(17374) [zypp] Exception.cc(log):166 MediaProducts.h(productsInMedia):103 CAUGHT:   File './media.1/products' not found on medium 'http://download.suse.de/ibs/SUSE:/Maintenance:/24571/SUSE_Updates_SLE-SERVER_12-SP5_s390x/'
2022-08-16 18:59:21 <3> install(17374) [zypp] MediaCurl.cc(doGetFileCopyFile):1575 curl error: 22: The requested URL returned error: 404 Not Found, temp file size 0 bytes.

Who can help with this issue?

Actions
Copy link
 #5

Updated by tjyrinki_suse over 1 year ago

Buildops can now reproduce the apache timeouts that cause this.

Actions
Copy link
 #6

Updated by rfan1 over 1 year ago

  • Related to action #115406: [qe-core] needle fails on add_update_test_repo added
Actions
Copy link
 #7

Updated by andriinikitin over 1 year ago

  • Status changed from New to Feedback

tjyrinki_suse wrote:

Buildops can now reproduce the apache timeouts that cause this.

Apparently the incoming load on download.suse.de increased at least twice comparing to recent weeks (maybe find why?), so the first explanation was that machine's resources must be scaled as well.
After adding more CPU power and reboot Apache is back to ultra fast mode.

I will keep monitoring, but let's consider that the issue is resolved and let me know if anything.

Actions
Copy link
 #8

Updated by rfan1 over 1 year ago

The issue is gone, thanks much @andriinikitin, can you please assign this ticket to yourself and close it?
BR//Richard.

Actions
Copy link
 #9

Updated by rfan1 over 1 year ago

  • Status changed from Feedback to Resolved
  • Assignee set to andriinikitin
  • % Done changed from 0 to 100
Actions
Copy link

Also available in: Atom PDF