openSUSE admin

Not sure why this is marked as "Private". Can someone change it to "public" pls.

vkrevs wrote in #note-1:

Not sure why this is marked as "Private". Can someone change it to "public" pls.

Looks like the issue has been resolved. Thank you!

Hi, I am having this issue as well.

I haven't been able to do the normal update for at least a few weeks now.

Help

rm -rf /var/cache/zypp/*
zsh: sure you want to delete all 4 files in /var/cache/zypp [yn]? y
[root@D5280:x86_64-Linux5:/home/jason]# zypper refresh                                                                           
Retrieving repository 'repo-non-oss (15.5)' metadata ......................................................................................................................................................................................................................................................[done]
Building repository 'repo-non-oss (15.5)' cache ...........................................................................................................................................................................................................................................................[done]
Retrieving repository 'repo-openh264 (15.5)' metadata .....................................................................................................................................................................................................................................................[done]
Building repository 'repo-openh264 (15.5)' cache ..........................................................................................................................................................................................................................................................[done]
Retrieving repository 'repo-oss (15.5)' metadata ..........................................................................................................................................................................................................................................................[done]
Building repository 'repo-oss (15.5)' cache ...............................................................................................................................................................................................................................................................[done]
Retrieving repository 'update-backports (15.5)' metadata ..................................................................................................................................................................................................................................................[done]
Building repository 'update-backports (15.5)' cache .......................................................................................................................................................................................................................................................[done]
Retrieving repository 'update-non-oss (15.5)' metadata ....................................................................................................................................................................................................................................................[done]
Building repository 'update-non-oss (15.5)' cache .........................................................................................................................................................................................................................................................[done]
Retrieving repository 'update-oss (15.5)' metadata ........................................................................................................................................................................................................................................................[done]
Building repository 'update-oss (15.5)' cache .............................................................................................................................................................................................................................................................[done]
Retrieving repository 'openSUSE:update-sle' metadata .....................................................................................................................................................................................................................................................[error]
Repository 'openSUSE:update-sle' is invalid.
[openSUSE:update-sle|http://cdn.opensuse.org/update/leap/15.5/sle] Valid metadata not found at specified URL
History:
 - File './repodata/4eaf6d2cf075569d872a18074dee79d4a12c64bae6d5e481d42ca5a388c07cdb-deltainfo.xml.gz' not found on medium 'http://cdn.opensuse.org/update/leap/15.5/sle'

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE:update-sle' because of the above error.
Retrieving repository 'Online updates for openSUSE Leap 15.5 (standard)' metadata .........................................................................................................................................................................................................................[done]
Building repository 'Online updates for openSUSE Leap 15.5 (standard)' cache ..............................................................................................................................................................................................................................[done]
Retrieving repository 'packman' metadata ..................................................................................................................................................................................................................................................................[done]
Building repository 'packman' cache .......................................................................................................................................................................................................................................................................[done]
Retrieving repository 'Update repository of openSUSE Backports' metadata ..................................................................................................................................................................................................................................[done]
Building repository 'Update repository of openSUSE Backports' cache .......................................................................................................................................................................................................................................[done]
Retrieving repository 'Non-OSS Repository' metadata .......................................................................................................................................................................................................................................................[done]
Building repository 'Non-OSS Repository' cache ............................................................................................................................................................................................................................................................[done]
Retrieving repository 'Open H.264 Codec (openSUSE Leap)' metadata .........................................................................................................................................................................................................................................[done]
Building repository 'Open H.264 Codec (openSUSE Leap)' cache ..............................................................................................................................................................................................................................................[done]
Retrieving repository 'Main Repository' metadata ..........................................................................................................................................................................................................................................................[done]
Building repository 'Main Repository' cache ...............................................................................................................................................................................................................................................................[done]
Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata .............................................................................................................................................................................................................[done]
Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ..................................................................................................................................................................................................................[done]
Retrieving repository 'Main Update Repository' metadata ...................................................................................................................................................................................................................................................[done]
Building repository 'Main Update Repository' cache ........................................................................................................................................................................................................................................................[done]
Retrieving repository 'Update Repository (Non-Oss)' metadata ..............................................................................................................................................................................................................................................[done]
Building repository 'Update Repository (Non-Oss)' cache ...................................................................................................................................................................................................................................................[done]
Retrieving repository 'snappy' metadata ...................................................................................................................................................................................................................................................................[done]
Building repository 'snappy' cache ........................................................................................................................................................................................................................................................................[done]
Some of the repositories have not been refreshed because of an error.

Hi, after forcibly removing the 15.5 update-sle repo, (and then having appear back magically), it now seems to be fixed.

I am still somewhat suspicious -- are you guys sure you guys haven't been hacked?
Obviously, a successful hijacking of an OS update service would be a great way to screw people over for fun and profit.......

Thank you.

p.s.
(Please do not dismiss my question out of hand without due diligence.)

p.p.s. I had the update outage from about late March 2024 to about 3 days ago.
I had assumed that it was a glitch at first, and ignored updating, until about a week ago when I decided to manually update, and ran into the issue repeatedly.
That seems to be an inordinately long time for a "glitch"

devzzzero wrote in #note-6:

I am still somewhat suspicious -- are you guys sure you guys haven't been hacked?
Obviously, a successful hijacking of an OS update service would be a great way to screw people over for fun and profit.......

It is possible that a mirror might be hacked / provide corrupted files, etc. But zypper and rpm systems have internal integrity checks of packages, which will catch that.

In other words - compromised download infrastructure will not lead to compromised target systems.

You can read more in this paragraph that I added to the wiki recently:

https://en.opensuse.org/openSUSE:Mirrors#Security_information_about_using_mirrors

p.p.s. I had the update outage from about late March 2024 to about 3 days ago.
I had assumed that it was a glitch at first, and ignored updating, until about a week ago when I decided to manually update, and ran into the issue repeatedly.
That seems to be an inordinately long time for a "glitch"

Again - we need to look at /var/log/zypper.log for problem days to be able to comment about details.

Thank you. I emailed the logs to andrii.nikitin @ suse.com

vkrevs wrote:

• File './repodata/432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz' not found on medium 'http://cdn.opensuse.org/update/leap/15.5/sle'

Another case was reported on slack and it looks I have identified root cause: repomd.xml is rendered by Apache instead of MirrorCache and it sets incorrect cache control flags, so repomd.xml is cached on cdn longer than needed.

I plan to fix it until tomorrow evening

andriinikitin wrote in #note-12:

It was identified that Apache on download.opensuse.org was miscofigured , which lead to incorrect cache instructions in the response headers for repomd.xml files.
Which lead to situations when repomd* files were cached on CDN inconsistently and longer than expected.
If am closing the ticket for now.

Still happening unfortunately:

`$ zypper lr --uri | grep update-backports
49 | openSUSE:update-backports | update-backports (15.5) | Yes | (r ) Yes | Yes | http://cdn.opensuse.org/update/leap/15.5/backports
50 | openSUSE:update-backports-debug | update-backports-debug (15.5) | Yes | (r ) Yes | Yes | http://cdn.opensuse.org/update/leap/15.5/backports_debug

Checking whether to refresh metadata for update-backports (15.5)
Retrieving: repomd.xml...................................[done]
Retrieving: repomd.xml...................................[done]
Retrieving: repomd.xml.asc...............................[done (827 B/s)]
Retrieving: repomd.xml.key...............................[done (923 B/s)]
Retrieving: repomd.xml ..................................[done]
Repository: update-backports (15.5)
Key Fingerprint: F044 C2C5 07A1 262B 538A AADD 8A49 EB03 25DB 7AE0
Key Name: openSUSE:Backports OBS Project openSUSE:Backports@build.opensuse.org
Key Algorithm: RSA 4096
Key Created: Wed May 10 15:46:12 2023
Key Expires: Sun May 9 15:46:12 2027
Rpm Name: gpg-pubkey-25db7ae0-645bae34
Signature verification failed for file 'repomd.xml' from repository 'update-backports (15.5)'.

Note: Signing data enables the recipient to verify that no modifications occurred after the data
were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
and in extreme cases even to a system compromise.

Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
whole repo.

Warning: This file was modified after it has been signed. This may have been a malicious change,
so it might not be trustworthy anymore! You should not continue unless you know it's safe.

Note: This might be a transient issue if the server is in the midst of receiving new data. The
data file and its signature are two files which must fit together. In case the request hit the
server in the midst of updating them, the signature verification might fail. After a few
minutes, when the server has updated its data, it should work again.

Signature verification failed for file 'repomd.xml' from repository 'update-backports (15.5)'. Continue? yes/no: yes
Retrieving: c0384b40987a12a9f10c6ffaa491edd3277ce92aa01e57c05169509f49399138-deltainfo.xml.gz .............[not found]
Retrieving repository 'update-backports (15.5)' metadata ..................................................[error]
Repository 'update-backports (15.5)' is invalid.
[openSUSE:update-backports|http://cdn.opensuse.org/update/leap/15.5/backports] Valid metadata not found at specified URL
History:

• File './repodata/c0384b40987a12a9f10c6ffaa491edd3277ce92aa01e57c05169509f49399138-deltainfo.xml.gz' not found on medium 'http://cdn.opensuse.org/update/leap/15.5/backports'

Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'update-backports (15.5)' because of the above error.`

I solved the issue by discontinuing use of cdn URLs.

vkrevs wrote in #note-13:

Still happening unfortunately:

If it happens again - is it persistent happening for hours or transient and resolves in few minutes?
I assume it is transient, but need the confirmation.

The failure usually lasts for couple of days at least.

Sent with Proton Mail secure email.

On Monday, June 3rd, 2024 at 3:40 AM, andriinikitin redmine@opensuse.org wrote:

[openSUSE Tracker]
Issue #159108 has been updated by andriinikitin.

vkrevs wrote in #note-13:

Still happening unfortunately:

If it happens again - is it persistent happening for hours or transient and resolves in few minutes?
I assume it is transient, but need the confirmation.

---

communication #159108: Repository 'update-sle (15.5)' is invalid.
https://progress.opensuse.org/issues/159108#change-804161

• Author: vkrevs
• Status: New
• Priority: Normal
• Assignee: andriinikitin
• Category: Mirrors

* Start date: 2024-04-17¶

$ zypper lr --uri | grep update-sle
55 | openSUSE:update-sle | update-sle (15.5) | Yes | (r ) Yes | Yes | http://cdn.opensuse.org/update/leap/15.5/sle

Doing a "zypper -v up -d" results in the following. Deleting the contents of /var/cache/zypp does not help. This has been the case for at least 2 days now.
...
Checking whether to refresh metadata for update-sle (15.5)
Retrieving: repomd.xml ...............................................................................................................................................................................................................................................................[done (916 B/s)]
Retrieving: repomd.xml.asc ...........................................................................................................................................................................................................................................................[done (481 B/s)]
Retrieving: repomd.xml.key ...........................................................................................................................................................................................................................................................[done (924 B/s)]
Retrieving: repomd.xml .........................................................................................................................................................................................................................................................................[done]
Repository: update-sle (15.5)
Key Fingerprint: FEAB 5025 39D8 46DB 2C09 61CA 70AF 9E81 39DB 7C82
Key Name: SuSE Package Signing Key build@suse.de

Key Algorithm: RSA 2048
Key Created: Mon Sep 21 09:21:47 2020
Key Expires: Fri Sep 20 09:21:47 2024
Rpm Name: gpg-pubkey-39db7c82-5f68629b
Signature verification failed for file 'repomd.xml' from repository 'update-sle (15.5)'.

Note: Signing data enables the recipient to verify that no modifications occurred after the data
were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
and in extreme cases even to a system compromise.

Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
whole repo.

Warning: This file was modified after it has been signed. This may have been a malicious change,
so it might not be trustworthy anymore! You should not continue unless you know it's safe.

Note: This might be a transient issue if the server is in the midst of receiving new data. The
data file and its signature are two files which must fit together. In case the request hit the
server in the midst of updating them, the signature verification might fail. After a few
minutes, when the server has updated its data, it should work again.

Signature verification failed for file 'repomd.xml' from repository 'update-sle (15.5)'. Continue? yes/no: yes
Retrieving: 432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz .............................................................................................................................................................................................[not found]
Retrieving repository 'update-sle (15.5)' metadata ............................................................................................................................................................................................................................................[error]
Repository 'update-sle (15.5)' is invalid.
[openSUSE:update-sle|http://cdn.opensuse.org/update/leap/15.5/sle] Valid metadata not found at specified URL
History:

• File './repodata/432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz' not found on medium 'http://cdn.opensuse.org/update/leap/15.5/sle'
• Can't provide ./repodata/432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz

Please check if the URIs defined for this repository are pointing to a valid repository.

---Files--------------------------------
summary.log (80.3 KB)

--
You have received this notification because you either subscribed to or are involved in this discussion.
To change your notification preferences, please visit https://progress.opensuse.org/my/account.

andriinikitin wrote in #note-17:

vkrevs wrote in #note-13:

Still happening unfortunately:

If it happens again - is it persistent happening for hours or transient and resolves in few minutes?
I assume it is transient, but need the confirmation.

Sadly, after the error ocurs, it never resolves in several minutes... usually hours or at least a day before it's working again.

vkrevs wrote in #note-20:

Sadly, after the error ocurs, it never resolves in several minutes... usually hours or at least a day before it's working again.

Could you please provide relevant fragment from /var/log/zypper.log demonstrating the issue that the problem occurs for an hour or at least dozen minutes?
(after May, 22)