openSUSE admin

rfan1 wrote:

This sporadic issue can be seen often in past few days, I am wondering if any issue with the repository itself.
Can I ask for some experts' investigation on it?

What does "temporarily unaccessible" mean? I think it would be useful to add some details about that.

pjessen wrote:

rfan1 wrote:

This sporadic issue can be seen often in past few days, I am wondering if any issue with the repository itself.
Can I ask for some experts' investigation on it?

What does "temporarily unaccessible" mean? I think it would be useful to add some details about that.

Sure,
Let me take one failed case as an example https://openqa.opensuse.org/tests/3051323#step/updates_packagekit_gpk/26.
Please see attached file.

I'm sorry, but we simply do not have sufficient information to even attempt to diagnose the issue. As I said above, to me "temporarily unavailable" suggests a time-out, but that is a client setting. It is certainly possible that download.o.o shows poor performance at times.

rfan1 wrote:

So is it possible to increase some hardware resource for the web server for http://download.opensuse.org? then we can get better performance than now.

It's certainly possible, but we don't know if http://download.o.o is causing the issue. Requests to download.o.o are mostly redirected to other mirrors. By the way, pontifex has 16 CPUs and 80Gb RAM.

To make any progress, I think we need two things:

a) which mirror is being tried?
b) what exactly does "temporarily unaccessible" mean?

When I look up the list of mirrors for that specific package, I see 14 in Germany:

    mirror1.hs-esslingen.de (DE)
    ftp.uni-kl.de (DE)
    ftp.rz.uni-wuerzburg.de (DE)
    mirror.de.leaseweb.net (DE)
    ftp.uni-erlangen.de (DE)
    ftp.uni-bayreuth.de (DE)
    ftp.halifax.rwth-aachen.de (DE)
    mirror.braun-software-solutions.de (DE)
    opensuse.schlundtech.de (DE)
    opensuse.mirror.iphh.net (DE)
    ftp.tu-chemnitz.de (DE)
    ftp.gwdg.de (DE)
    suse.uni-leipzig.de (DE)
    mirror.linux-schulserver.de (DE)

Thanks much!
I can provide the installation logs here, and please see the attached file for more detail logs, some of the repo mirrors can be seen in the log file.

At the same time, if the file can't provide the enough information you need, I will ask for our developer team to check if we can add some debug messages.

23-02-08 19:23:44 <1> susetest(3229) [zypp-core] PathInfo.cc(chmod):1098 {T:140341557425920} assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600
2023-02-08 19:23:44 <2> susetest(3229) [zypp::media] MediaMultiCurl.cc(run):1151 {T:140341557425920} overall result
2023-02-08 19:23:44 <2> susetest(3229) [zypp::media] MediaMultiCurl.cc(run):1155 {T:140341557425920} #0: state: 4 received: 318688 url: http://mirror.braun-software-solutions.de/opensuse/update/leap/15.4/sle/x86_64/ruby2.5-2.5.9-150000.4.26.1.x86_64.rpm
2023-02-08 19:23:44 <2> susetest(3229) [zypp::media] MediaMultiCurl.cc(run):1155 {T:140341557425920} #1: state: 3 received: 0 url: http://ftp.tu-chemnitz.de/pub/linux/opensuse/update/leap/15.4/sle/x86_64/ruby2.5-2.5.9-150000.4.26.1.x86_64.rpm
2023-02-08 19:23:44 <2> susetest(3229) [zypp::media] MediaMultiCurl.cc(run):1155 {T:140341557425920} #2: state: 5 received: 56544 url: http://opensuse.schlundtech.de//update/leap/15.4/sle/x86_64/ruby2.5-2.5.9-150000.4.26.1.x86_64.rpm
2023-02-08 19:23:44 <2> susetest(3229) [zypp::media] MediaMultiCurl.cc(run):1155 {T:140341557425920} #3: state: 3 received: 131072 url: http://ftp.gwdg.de/pub/opensuse/update/leap/15.4/sle/x86_64/ruby2.5-2.5.9-150000.4.26.1.x86_64.rpm
2023-02-08 19:23:44 <2> susetest(3229) [zypp::media] MediaMultiCurl.cc(run):1155 {T:140341557425920} #4: state: 3 received: 116991 url: http://mirror.de.leaseweb.net/opensuse/update/leap/15.4/sle/x86_64/ruby2.5-2.5.9-150000.4.26.1.x86_64.rpm

2023-02-08 19:23:44 <1> susetest(3229) [zypp] PackageProvider.cc(providePackage):412 {T:140341557425920} provide Package (114386)libldap-2_4-2-2.4.46-150200.14.11.2.x86_64(repo-sle-update)
2023-02-08 19:23:44 <1> susetest(3229) [packagekit] ../backends/zypp/pk-backend-zypp.cpp(start):389 {T:140341557425920} (114386)libldap-2_4-2-2.4.46-150200.14.11.2.x86_64(repo-sle-update) http://download.opensuse.org/update/leap/15.4/sle/
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media] RepoProvideFile.cc(provideFile):241 {T:140341557425920} [1]./x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm{167.5 KiB|sha256-fd72b530ddbfd9c94b1afd44d1db652e4b877df377622a43dc806586b4787555|}
2023-02-08 19:23:44 <1> susetest(3229) [zypp::fetcher++] Fetcher.cc(addCachePath):331 {T:140341557425920} Adding fetcher cache: '/var/cache/zypp/packages/repo-sle-update'.
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media] RepoProvideFile.cc(provideFile):260 {T:140341557425920} Added cache path /var/cache/zypp/packages/repo-sle-update
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media] RepoProvideFile.cc(provideFile):294 {T:140341557425920} Providing file of repo 'repo-sle-update' from http://download.opensuse.org/update/leap/15.4/sle/
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media++] RepoProvideFile.cc(setVerifierForRepo):203 {T:140341557425920} No media verifier for repo 'repo-sle-update' media.1/media does not exist in '/var/cache/zypp/raw/repo-sle-update'
2023-02-08 19:23:44 <1> susetest(3229) [zypp::fetcher] Fetcher.cc(downloadAndReadIndexList):705 {T:140341557425920} No indexes to read.
2023-02-08 19:23:44 <1> susetest(3229) [zypp::fetcher] Fetcher.cc(locateInCache):362 {T:140341557425920} start fetcher with 1 cache directories.
2023-02-08 19:23:44 <1> susetest(3229) [zypp::fetcher] Fetcher.cc(provideToDest):533 {T:140341557425920} Not found in cache, retrieving...
2023-02-08 19:23:44 <1> susetest(3229) [zypp::fetcher++] MediaSetAccess.cc(provide):278 {T:140341557425920} Going to try to provide  file ./x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm from media number 1
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media++] MediaManager.cc(checkDesired):155 {T:140341557425920} checkDesired(23): desired (cached)
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1336 {T:140341557425920} dest: /var/tmp/AP_0x5j27xl/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1337 {T:140341557425920} temp: /var/tmp/AP_0x5j27xl/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm.new.zypp.hK7M1k
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1173 {T:140341557425920} ./x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm
2023-02-08 19:23:44 <1> susetest(3229) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1183 {T:140341557425920} URL: http://download.opensuse.org/update/leap/15.4/sle/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm
2023-02-08 19:24:14 <3> susetest(3229) [zypp::media] MediaCurl.cc(doGetFileCopyFile):1248 {T:140341557425920} curl error: 22: The requested URL returned error: 502, temp file size 0 bytes.
2023-02-08 19:24:14 <5> susetest(3229) [zypp-core] Exception.cc(log):186 {T:140341557425920} MediaCurl.cc(evaluateCurlCode):792 THROW:    Location 'http://download.opensuse.org/update/leap/15.4/sle/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm' is temporarily unaccessible.
2023-02-08 19:24:14 <5> susetest(3229) [zypp-core] Exception.cc(log):186 {T:140341557425920} MediaCurl.cc(doGetFileCopyFile):1264 RETHROW:  Location 'http://download.opensuse.org/update/leap/15.4/sle/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm' is temporarily unaccessible.
2023-02-08 19:24:14 <5> susetest(3229) [zypp-core] Exception.cc(log):186 {T:140341557425920} MediaMultiCurl.cc(doGetFileCopy):1365 RETHROW:  Location 'http://download.opensuse.org/update/leap/15.4/sle/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm' is temporarily unaccessible.
2023-02-08 19:24:14 <1> susetest(3229) [zypp-core] PathInfo.cc(unlink):706 {T:140341557425920} unlink /var/tmp/AP_0x5j27xl/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm.new.zypp.hK7M1k
2023-02-08 19:24:14 <5> susetest(3229) [zypp-core] Exception.cc(log):186 {T:140341557425920} MediaCurl.cc(getFileCopy):693 RETHROW:  Location 'http://download.opensuse.org/update/leap/15.4/sle/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm' is temporarily unaccessible.
2023-02-08 19:24:14 <5> susetest(3229) [zypp-core] Exception.cc(log):186 {T:140341557425920} MediaSetAccess.cc(provide):288 CAUGHT:   Location 'http://download.opensuse.org/update/leap/15.4/sle/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm' is temporarily unaccessible.
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media++] MediaHandler.cc(getDetectedDevices):1347 {T:140341557425920} No devices for this medium
2023-02-08 19:24:14 <1> susetest(3229) [zypp::fetcher++] MediaSetAccess.cc(provide):298 {T:140341557425920} Media couldn't provide file ./x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm , releasing.
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media++] MediaManager.cc(release):508 {T:140341557425920} release(id=23)
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media++] MediaHandler.cc(release):713 {T:140341557425920} Request to release attached media http<http://download.opensuse.org/update/leap/15.4/sle/>, use count=1
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media++] MediaHandler.cc(release):720 {T:140341557425920} Releasing media http<http://download.opensuse.org/update/leap/15.4/sle/>
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media] MediaHandler.cc(disconnect):692 {T:140341557425920} Disconnected: http://download.opensuse.org/update/leap/15.4/sle/ attached; localRoot "/var/tmp/AP_0x5j27xl"
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media++] MediaHandler.cc(removeAttachPoint):149 {T:140341557425920} MediaHandler - checking if to remove attach point
2023-02-08 19:24:14 <1> susetest(3229) [zypp-core] PathInfo.cc(recursive_rmdir):434 {T:140341557425920} recursive_rmdir /var/tmp/AP_0x5j27xl 
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media] MediaHandler.cc(removeAttachPoint):161 {T:140341557425920} Deleted default attach point /var/tmp/AP_0x5j27xl
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media] MediaHandler.cc(release):774 {T:140341557425920} Released: http://download.opensuse.org/update/leap/15.4/sle/ not attached; localRoot ""
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media] MediaManager.cc(releaseAll):546 {T:140341557425920} Releasing all attached media
2023-02-08 19:24:14 <1> susetest(3229) [zypp::media++] MediaManager.cc(operator()):552 {T:140341557425920} Releasing media id 22

rfan1 wrote:

2023-02-08 19:23:44 <1> susetest(3229) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1183 {T:140341557425920} URL: http://download.opensuse.org/update/leap/15.4/sle/x86_64/libldap-2_4-2-2.4.46-150200.14.11.2.x86_64.rpm
2023-02-08 19:24:14 <3> susetest(3229) [zypp::media] MediaCurl.cc(doGetFileCopyFile):1248 {T:140341557425920} curl error: 22: The requested URL returned error: 502, temp file size 0 bytes.

That is the problem - HTTP 502. That is a mirrorcache issue, I suspect. Assigning to Andrii.

HTTP code 502 means that service is not available.

Occasional downtime happen on download.opensuse.org - e.g. recently there were cases of filesystem corruption and DB overload.

We work on improving overall experience, but considering limited resources - not sure what else can be done about this ticket.