Project

General

Profile

Actions
Copy link

communication #159108

open

Repository 'update-sle (15.5)' is invalid.

Added by vkrevs 8 months ago. Updated about 2 months ago.

Status:
Feedback
Priority:
Normal
Assignee:
andriinikitin
Category:
Mirrors
Target version:
-
Start date:
2024-04-17
Due date:
% Done:

0%

Estimated time:

Description

$ zypper lr --uri | grep update-sle
55 | openSUSE:update-sle | update-sle (15.5) | Yes | (r ) Yes | Yes | http://cdn.opensuse.org/update/leap/15.5/sle

Doing a "zypper -v up -d" results in the following. Deleting the contents of /var/cache/zypp does not help. This has been the case for at least 2 days now.
...
Checking whether to refresh metadata for update-sle (15.5)
Retrieving: repomd.xml ...............................................................................................................................................................................................................................................................[done (916 B/s)]
Retrieving: repomd.xml.asc ...........................................................................................................................................................................................................................................................[done (481 B/s)]
Retrieving: repomd.xml.key ...........................................................................................................................................................................................................................................................[done (924 B/s)]
Retrieving: repomd.xml .........................................................................................................................................................................................................................................................................[done]
Repository: update-sle (15.5)
Key Fingerprint: FEAB 5025 39D8 46DB 2C09 61CA 70AF 9E81 39DB 7C82
Key Name: SuSE Package Signing Key build@suse.de
Key Algorithm: RSA 2048
Key Created: Mon Sep 21 09:21:47 2020
Key Expires: Fri Sep 20 09:21:47 2024
Rpm Name: gpg-pubkey-39db7c82-5f68629b
Signature verification failed for file 'repomd.xml' from repository 'update-sle (15.5)'.

Note: Signing data enables the recipient to verify that no modifications occurred after the data
were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
and in extreme cases even to a system compromise.

Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
whole repo.

Warning: This file was modified after it has been signed. This may have been a malicious change,
so it might not be trustworthy anymore! You should not continue unless you know it's safe.

Note: This might be a transient issue if the server is in the midst of receiving new data. The
data file and its signature are two files which must fit together. In case the request hit the
server in the midst of updating them, the signature verification might fail. After a few
minutes, when the server has updated its data, it should work again.

Signature verification failed for file 'repomd.xml' from repository 'update-sle (15.5)'. Continue? yes/no: yes
Retrieving: 432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz .............................................................................................................................................................................................[not found]
Retrieving repository 'update-sle (15.5)' metadata ............................................................................................................................................................................................................................................[error]
Repository 'update-sle (15.5)' is invalid.
[openSUSE:update-sle|http://cdn.opensuse.org/update/leap/15.5/sle] Valid metadata not found at specified URL
History:

  • File './repodata/432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz' not found on medium 'http://cdn.opensuse.org/update/leap/15.5/sle'
  • Can't provide ./repodata/432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz

Please check if the URIs defined for this repository are pointing to a valid repository.

Files

Download all files
summary.log (80.3 KB) summary.log andriinikitin, 2024-05-07 11:28
zypper.log.20240611 (1.28 MB) zypper.log.20240611 vkrevs, 2024-06-11 09:13

Related issues 1 (1 open — 0 closed)

Related to openSUSE admin - communication #161078: Leap 15.5 repodata issues on CDNNew2024-05-28

Actions
Actions
Copy link
 #1

Updated by vkrevs 8 months ago

Not sure why this is marked as "Private". Can someone change it to "public" pls.

Actions
Copy link
 #2

Updated by crameleon 8 months ago

  • Category set to Mirrors
  • Private changed from Yes to No
Actions
Copy link
 #3

Updated by vkrevs 8 months ago

vkrevs wrote in #note-1:

Not sure why this is marked as "Private". Can someone change it to "public" pls.

Looks like the issue has been resolved. Thank you!

Actions
Copy link
 #4

Updated by devzzzero 8 months ago

Hi, I am having this issue as well.

I haven't been able to do the normal update for at least a few weeks now.

Help

rm -rf /var/cache/zypp/*
zsh: sure you want to delete all 4 files in /var/cache/zypp [yn]? y
[root@D5280:x86_64-Linux5:/home/jason]# zypper refresh                                                                           
Retrieving repository 'repo-non-oss (15.5)' metadata ......................................................................................................................................................................................................................................................[done]
Building repository 'repo-non-oss (15.5)' cache ...........................................................................................................................................................................................................................................................[done]
Retrieving repository 'repo-openh264 (15.5)' metadata .....................................................................................................................................................................................................................................................[done]
Building repository 'repo-openh264 (15.5)' cache ..........................................................................................................................................................................................................................................................[done]
Retrieving repository 'repo-oss (15.5)' metadata ..........................................................................................................................................................................................................................................................[done]
Building repository 'repo-oss (15.5)' cache ...............................................................................................................................................................................................................................................................[done]
Retrieving repository 'update-backports (15.5)' metadata ..................................................................................................................................................................................................................................................[done]
Building repository 'update-backports (15.5)' cache .......................................................................................................................................................................................................................................................[done]
Retrieving repository 'update-non-oss (15.5)' metadata ....................................................................................................................................................................................................................................................[done]
Building repository 'update-non-oss (15.5)' cache .........................................................................................................................................................................................................................................................[done]
Retrieving repository 'update-oss (15.5)' metadata ........................................................................................................................................................................................................................................................[done]
Building repository 'update-oss (15.5)' cache .............................................................................................................................................................................................................................................................[done]
Retrieving repository 'openSUSE:update-sle' metadata .....................................................................................................................................................................................................................................................[error]
Repository 'openSUSE:update-sle' is invalid.
[openSUSE:update-sle|http://cdn.opensuse.org/update/leap/15.5/sle] Valid metadata not found at specified URL
History:
 - File './repodata/4eaf6d2cf075569d872a18074dee79d4a12c64bae6d5e481d42ca5a388c07cdb-deltainfo.xml.gz' not found on medium 'http://cdn.opensuse.org/update/leap/15.5/sle'

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'openSUSE:update-sle' because of the above error.
Retrieving repository 'Online updates for openSUSE Leap 15.5 (standard)' metadata .........................................................................................................................................................................................................................[done]
Building repository 'Online updates for openSUSE Leap 15.5 (standard)' cache ..............................................................................................................................................................................................................................[done]
Retrieving repository 'packman' metadata ..................................................................................................................................................................................................................................................................[done]
Building repository 'packman' cache .......................................................................................................................................................................................................................................................................[done]
Retrieving repository 'Update repository of openSUSE Backports' metadata ..................................................................................................................................................................................................................................[done]
Building repository 'Update repository of openSUSE Backports' cache .......................................................................................................................................................................................................................................[done]
Retrieving repository 'Non-OSS Repository' metadata .......................................................................................................................................................................................................................................................[done]
Building repository 'Non-OSS Repository' cache ............................................................................................................................................................................................................................................................[done]
Retrieving repository 'Open H.264 Codec (openSUSE Leap)' metadata .........................................................................................................................................................................................................................................[done]
Building repository 'Open H.264 Codec (openSUSE Leap)' cache ..............................................................................................................................................................................................................................................[done]
Retrieving repository 'Main Repository' metadata ..........................................................................................................................................................................................................................................................[done]
Building repository 'Main Repository' cache ...............................................................................................................................................................................................................................................................[done]
Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata .............................................................................................................................................................................................................[done]
Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ..................................................................................................................................................................................................................[done]
Retrieving repository 'Main Update Repository' metadata ...................................................................................................................................................................................................................................................[done]
Building repository 'Main Update Repository' cache ........................................................................................................................................................................................................................................................[done]
Retrieving repository 'Update Repository (Non-Oss)' metadata ..............................................................................................................................................................................................................................................[done]
Building repository 'Update Repository (Non-Oss)' cache ...................................................................................................................................................................................................................................................[done]
Retrieving repository 'snappy' metadata ...................................................................................................................................................................................................................................................................[done]
Building repository 'snappy' cache ........................................................................................................................................................................................................................................................................[done]
Some of the repositories have not been refreshed because of an error.
Actions
Copy link
 #5

Updated by andriinikitin 8 months ago ยท Edited

  • Status changed from New to Feedback
  • Assignee set to andriinikitin

In my understanding the asynchronous workflow of the infrastructure (OBS publishing, syncing to openSUSE mirrors, and download redirector scanner) does not guarantee absence of occasional errors related to concurrent changes.
Especially for the projects that are published often and being requested by many users.
Now, caching at CDN side probably contributes to the problem to some extend.

So, first recommendation will be: just retry in a few minutes and see if the problem is resolved.

If that doesn't help or if you still want us to look into the root cause - please provide corresponding fragment of timeline from /var/log/zypper.log , it should have more explanation e.g. about timing and exact url(s) in use, status codes, etc.

Regards,
Andrii Nikitin

Actions
Copy link
 #6

Updated by devzzzero 8 months ago

Hi, after forcibly removing the 15.5 update-sle repo, (and then having appear back magically), it now seems to be fixed.

I am still somewhat suspicious -- are you guys sure you guys haven't been hacked?
Obviously, a successful hijacking of an OS update service would be a great way to screw people over for fun and profit.......

Thank you.

p.s.
(Please do not dismiss my question out of hand without due diligence.)

p.p.s. I had the update outage from about late March 2024 to about 3 days ago.
I had assumed that it was a glitch at first, and ignored updating, until about a week ago when I decided to manually update, and ran into the issue repeatedly.
That seems to be an inordinately long time for a "glitch"

Actions
Copy link
 #7

Updated by andriinikitin 8 months ago

devzzzero wrote in #note-6:

I am still somewhat suspicious -- are you guys sure you guys haven't been hacked?
Obviously, a successful hijacking of an OS update service would be a great way to screw people over for fun and profit.......

It is possible that a mirror might be hacked / provide corrupted files, etc. But zypper and rpm systems have internal integrity checks of packages, which will catch that.

In other words - compromised download infrastructure will not lead to compromised target systems.

You can read more in this paragraph that I added to the wiki recently:

https://en.opensuse.org/openSUSE:Mirrors#Security_information_about_using_mirrors

p.p.s. I had the update outage from about late March 2024 to about 3 days ago.
I had assumed that it was a glitch at first, and ignored updating, until about a week ago when I decided to manually update, and ran into the issue repeatedly.
That seems to be an inordinately long time for a "glitch"

Again - we need to look at /var/log/zypper.log for problem days to be able to comment about details.

Actions
Copy link
 #8

Updated by devzzzero 8 months ago

Thank you. I emailed the logs to andrii.nikitin @ suse.com

Actions
Copy link
 #9

Updated by andriinikitin 8 months ago

devzzzero wrote in #note-8:

Thank you. I emailed the logs to andrii.nikitin @ suse.com

I've attached file summary.log captured with command: 

xzgrep -A1 -E 'Exiting main|Hi, me zypper|ABORT|Signature verification failed' zypper.log* | grep -v -- '--' > summary.log

It indicates no problem with updates for recent moths.
There was problem on 26-Apr with repo refresh during zypper search, but it resolved after retry in few minutes.

Actions
Copy link
 #10

Updated by andriinikitin 7 months ago

vkrevs wrote:

Another case was reported on slack and it looks I have identified root cause: repomd.xml is rendered by Apache instead of MirrorCache and it sets incorrect cache control flags, so repomd.xml is cached on cdn longer than needed.

I plan to fix it until tomorrow evening

Actions
Copy link
 #11

Updated by andriinikitin 7 months ago

  • Status changed from Feedback to In Progress
Actions
Copy link
 #12

Updated by andriinikitin 7 months ago

  • Status changed from In Progress to Resolved

It was identified that Apache on download.opensuse.org was miscofigured , which lead to incorrect cache instructions in the response headers for repomd.xml files.
Which lead to situations when repomd* files were cached on CDN inconsistently and longer than expected.
If am closing the ticket for now.

Actions
Copy link
 #13

Updated by vkrevs 7 months ago

andriinikitin wrote in #note-12:

It was identified that Apache on download.opensuse.org was miscofigured , which lead to incorrect cache instructions in the response headers for repomd.xml files.
Which lead to situations when repomd* files were cached on CDN inconsistently and longer than expected.
If am closing the ticket for now.

Still happening unfortunately:

`$ zypper lr --uri | grep update-backports
49 | openSUSE:update-backports | update-backports (15.5) | Yes | (r ) Yes | Yes | http://cdn.opensuse.org/update/leap/15.5/backports
50 | openSUSE:update-backports-debug | update-backports-debug (15.5) | Yes | (r ) Yes | Yes | http://cdn.opensuse.org/update/leap/15.5/backports_debug

Checking whether to refresh metadata for update-backports (15.5)
Retrieving: repomd.xml...................................[done]
Retrieving: repomd.xml...................................[done]
Retrieving: repomd.xml.asc...............................[done (827 B/s)]
Retrieving: repomd.xml.key...............................[done (923 B/s)]
Retrieving: repomd.xml ..................................[done]
Repository: update-backports (15.5)
Key Fingerprint: F044 C2C5 07A1 262B 538A AADD 8A49 EB03 25DB 7AE0
Key Name: openSUSE:Backports OBS Project openSUSE:Backports@build.opensuse.org
Key Algorithm: RSA 4096
Key Created: Wed May 10 15:46:12 2023
Key Expires: Sun May 9 15:46:12 2027
Rpm Name: gpg-pubkey-25db7ae0-645bae34
Signature verification failed for file 'repomd.xml' from repository 'update-backports (15.5)'.

Note: Signing data enables the recipient to verify that no modifications occurred after the data
were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
and in extreme cases even to a system compromise.

Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
whole repo.

Warning: This file was modified after it has been signed. This may have been a malicious change,
so it might not be trustworthy anymore! You should not continue unless you know it's safe.

Note: This might be a transient issue if the server is in the midst of receiving new data. The
data file and its signature are two files which must fit together. In case the request hit the
server in the midst of updating them, the signature verification might fail. After a few
minutes, when the server has updated its data, it should work again.

Signature verification failed for file 'repomd.xml' from repository 'update-backports (15.5)'. Continue? yes/no: yes
Retrieving: c0384b40987a12a9f10c6ffaa491edd3277ce92aa01e57c05169509f49399138-deltainfo.xml.gz .............[not found]
Retrieving repository 'update-backports (15.5)' metadata ..................................................[error]
Repository 'update-backports (15.5)' is invalid.
[openSUSE:update-backports|http://cdn.opensuse.org/update/leap/15.5/backports] Valid metadata not found at specified URL
History:

Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'update-backports (15.5)' because of the above error.`

Actions
Copy link
 #14

Updated by crameleon 7 months ago

Actions
Copy link
 #15

Updated by devzzzero 7 months ago

  • Status changed from Resolved to New

Thank you!

Actions
Copy link
 #16

Updated by mrmazda 7 months ago

I solved the issue by discontinuing use of cdn URLs.

Actions
Copy link
 #17

Updated by andriinikitin 7 months ago

vkrevs wrote in #note-13:

Still happening unfortunately:

If it happens again - is it persistent happening for hours or transient and resolves in few minutes?
I assume it is transient, but need the confirmation.

Actions
Copy link
 #18

Updated by andriinikitin 7 months ago

  • Status changed from New to Feedback
Actions
Copy link
 #19

Updated by devzzzero 7 months ago

The failure usually lasts for couple of days at least.

Sent with Proton Mail secure email.

On Monday, June 3rd, 2024 at 3:40 AM, andriinikitin redmine@opensuse.org wrote:

[openSUSE Tracker]
Issue #159108 has been updated by andriinikitin.

vkrevs wrote in #note-13:

Still happening unfortunately:

If it happens again - is it persistent happening for hours or transient and resolves in few minutes?
I assume it is transient, but need the confirmation.

communication #159108: Repository 'update-sle (15.5)' is invalid.
https://progress.opensuse.org/issues/159108#change-804161

  • Author: vkrevs
  • Status: New
  • Priority: Normal
  • Assignee: andriinikitin
  • Category: Mirrors

* Start date: 2024-04-17

$ zypper lr --uri | grep update-sle
55 | openSUSE:update-sle | update-sle (15.5) | Yes | (r ) Yes | Yes | http://cdn.opensuse.org/update/leap/15.5/sle

Doing a "zypper -v up -d" results in the following. Deleting the contents of /var/cache/zypp does not help. This has been the case for at least 2 days now.
...
Checking whether to refresh metadata for update-sle (15.5)
Retrieving: repomd.xml ...............................................................................................................................................................................................................................................................[done (916 B/s)]
Retrieving: repomd.xml.asc ...........................................................................................................................................................................................................................................................[done (481 B/s)]
Retrieving: repomd.xml.key ...........................................................................................................................................................................................................................................................[done (924 B/s)]
Retrieving: repomd.xml .........................................................................................................................................................................................................................................................................[done]
Repository: update-sle (15.5)
Key Fingerprint: FEAB 5025 39D8 46DB 2C09 61CA 70AF 9E81 39DB 7C82
Key Name: SuSE Package Signing Key build@suse.de

Key Algorithm: RSA 2048
Key Created: Mon Sep 21 09:21:47 2020
Key Expires: Fri Sep 20 09:21:47 2024
Rpm Name: gpg-pubkey-39db7c82-5f68629b
Signature verification failed for file 'repomd.xml' from repository 'update-sle (15.5)'.

Note: Signing data enables the recipient to verify that no modifications occurred after the data
were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
and in extreme cases even to a system compromise.

Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
whole repo.

Warning: This file was modified after it has been signed. This may have been a malicious change,
so it might not be trustworthy anymore! You should not continue unless you know it's safe.

Note: This might be a transient issue if the server is in the midst of receiving new data. The
data file and its signature are two files which must fit together. In case the request hit the
server in the midst of updating them, the signature verification might fail. After a few
minutes, when the server has updated its data, it should work again.

Signature verification failed for file 'repomd.xml' from repository 'update-sle (15.5)'. Continue? yes/no: yes
Retrieving: 432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz .............................................................................................................................................................................................[not found]
Retrieving repository 'update-sle (15.5)' metadata ............................................................................................................................................................................................................................................[error]
Repository 'update-sle (15.5)' is invalid.
[openSUSE:update-sle|http://cdn.opensuse.org/update/leap/15.5/sle] Valid metadata not found at specified URL
History:

  • File './repodata/432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz' not found on medium 'http://cdn.opensuse.org/update/leap/15.5/sle'
  • Can't provide ./repodata/432eb59088755eb918c8b75cdb13917f37c7f11405d54afa8a673d235190f93d-deltainfo.xml.gz

Please check if the URIs defined for this repository are pointing to a valid repository.

---Files--------------------------------
summary.log (80.3 KB)

--
You have received this notification because you either subscribed to or are involved in this discussion.
To change your notification preferences, please visit https://progress.opensuse.org/my/account.

Actions
Copy link
 #20

Updated by vkrevs 7 months ago

andriinikitin wrote in #note-17:

vkrevs wrote in #note-13:

Still happening unfortunately:

If it happens again - is it persistent happening for hours or transient and resolves in few minutes?
I assume it is transient, but need the confirmation.

Sadly, after the error ocurs, it never resolves in several minutes... usually hours or at least a day before it's working again.

Actions
Copy link
 #21

Updated by andriinikitin 7 months ago

vkrevs wrote in #note-20:

Sadly, after the error ocurs, it never resolves in several minutes... usually hours or at least a day before it's working again.

Could you please provide relevant fragment from /var/log/zypper.log demonstrating the issue that the problem occurs for an hour or at least dozen minutes?
(after May, 22)

Actions
Copy link
 #22

Updated by vkrevs 6 months ago

andriinikitin wrote in #note-21:

vkrevs wrote in #note-20:

Sadly, after the error ocurs, it never resolves in several minutes... usually hours or at least a day before it's working again.

Could you please provide relevant fragment from /var/log/zypper.log demonstrating the issue that the problem occurs for an hour or at least dozen minutes?
(after May, 22)

Happening right now. Checked for updated at 09:43, then again at 10:09. Zypper log entries from today attached.

Actions
Copy link
 #23

Updated by andriinikitin about 2 months ago

Hi,

Sorry for delayed reply.

vkrevs wrote in #note-22:

Happening right now. Checked for updated at 09:43, then again at 10:09. Zypper log entries from today attached.

It tried to reproduce the problem locally. Then also set up some monitoring, which could explain the issue. But without any success.
Also was expecting some complains from others if that is the issue. Also not much.
It is possible that it was some temporary glitch or unlucky timing.

If this is still a problem - I must engage the guys who manages CDN. But it is better if you re-confirm that the problem is still happening and provide most recent log, so they have something fresh to work with.

Regards,
Andrii Nikitin

Actions
Copy link

Also available in: Atom PDF