Project

General

Profile

tickets #94273

All Leap 15.3 downloads are broken

Added by lkocman 11 months ago. Updated 11 months ago.

Status:
Closed
Priority:
Immediate
Assignee:
Category:
Mirrors
Target version:
-
Start date:
2021-06-19
Due date:
% Done:

0%

Estimated time:

Description

History

#1 Updated by andriinikitin 11 months ago

I think the the problem here is that the mirrors got new version, but scanners couldn't update the info for some reasons.

I am not sure what to do exactly and how to troubleshoot such problem, but I decided that at least scanning manually the problem servers should fix 404, so I did:

/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso widehat.opensuse.org
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso widehat.opensuse.org

The last command was just hanging, so I've looked in mirrordb2 and had to kill some stalled queries and command succeeded.
Should be fixed now, at least for most of countries, I will try to make sure that all mirrors from mirrorlist are rescanned for these locations

#2 Updated by pjessen 11 months ago

  • Tracker changed from communication to tickets
  • Subject changed from All Leap downloads are broken to All Leap 15.3 downloads are broken
  • Private changed from Yes to No

andriinikitin wrote:

I think the the problem here is that the mirrors got new version, but scanners couldn't update the info for some reasons.

I am not sure what to do exactly and how to troubleshoot such problem, but I decided that at least scanning manually the problem servers should fix 404, so I did:

/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso widehat.opensuse.org
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso widehat.opensuse.org

The last command was just hanging, so I've looked in mirrordb2 and had to kill some stalled queries and command succeeded.
Should be fixed now, at least for most of countries, I will try to make sure that all mirrors from mirrorlist are rescanned for these locations

We are certain this is in no way related to the linking of ftp-stage to ftp ? Scanning ought to be running continuously from olaf. It seems very odd if there is a problem with scanning that only affects Leap 15.3.

#3 Updated by andriinikitin 11 months ago

pjessen wrote:

We are certain this is in no way related to the linking of ftp-stage to ftp ? Scanning ought to be running continuously from olaf. It seems very odd if there is a problem with scanning that only affects Leap 15.3.

I think the problem is both with scanning and publishing of 15.3 iso, but it is hard to tell which exactly

#4 Updated by andriinikitin 11 months ago

Now I think I did understand the problem wrong from the start. Leap 15.3 was released already, so no new .iso must be published anymore, correct?
So the problem is that .iso got updated on mirrors, when it shouldn't happen anymore, correct?

#5 Updated by lkocman 11 months ago

It seems to work now for me

I could get e.g. without any issues (same for aarch64 and intel)
https://download.opensuse.org/distribution/leap/15.3/iso/openSUSE-Leap-15.3-DVD-ppc64le.iso

Same for links in http://get.opensuse.org/leap/#download

Lubos

#6 Updated by pjessen 11 months ago

andriinikitin wrote:

Now I think I did understand the problem wrong from the start. Leap 15.3 was released already, so no new .iso must be published anymore, correct?

That is correct. Looking at download.o.o/distribution/opensuse/leap/15.3/iso, they are all dated 26 May 13:36. On my mirror (Hostsuisse), the ISO images are dated 24 May 13:46 to 14:25. This corresponds to contents of ftp-stage whereas download.o.o is showing /srv/ftp.

So the problem is that .iso got updated on mirrors, when it shouldn't happen anymore, correct?

I don't know where mirrors would get new isos from - mirrors rsync from ftp-stage.
Looking at ftp.gwdg.de, the isos are dated May 24.

#7 Updated by andriinikitin 11 months ago

pjessen wrote:

Looking at ftp.gwdg.de, the isos are dated May 24.

Ah, then the problem is that most of mirrors now have "-Media" in their name, e.g. gwdg.de has:
openSUSE-Leap-15.3-DVD-x86_64-Media.iso
While download.o.o has:
openSUSE-Leap-15.3-DVD-x86_64.iso

And the the mirrors were updated few days ago recently, because mirrorbrain did have info about correct filename on the mirrors.

#8 Updated by andriinikitin 11 months ago

andriinikitin wrote:

And the the mirrors were updated few days ago recently, because mirrorbrain did have info about correct filename on the mirrors.

and in appliances the files are from June with Build9.96 http://ftp.gwdg.de/pub/opensuse/distribution/leap/15.3/appliances/iso/
while download.o.o has Build9.88 , so these were new releases of 15.3 - not sure what went wrong

#9 Updated by andriinikitin 11 months ago

  • Assignee set to lkocman

So it looks the reason for the problem is that Leap 15.3 was rebuilt in OBS (probably some of dependencies changed) :

> rsync obspublish::openqa/openSUSE:Leap:15.3/standard/x86_64/ | grep 2021/06
drwxr-xr-x          8,192 2021/06/18 09:18:50 .
-rw-r--r--         67,477 2021/06/18 09:15:10 :full.metacache
-rw-r--r--        555,409 2021/06/18 09:08:06 :full.solv
-rw-r--r--        228,828 2021/06/18 09:08:07 :full.useforbuild
-rw-r--r--      1,437,129 2021/06/18 09:18:50 :full.xcache
-rw-r--r--        371,165 2021/06/18 09:15:10 :packstatus
-rw-r--r--            123 2021/06/18 09:15:10 :schedulerstate

lkocman could you please completely disable building and publishing of Leap-15.3 project in OBS https://build.opensuse.org/repositories/openSUSE:Leap:15.3 ?
And I guess the same for Leap-15.3:Images project.
Then assign the ticket back to me and I will make sure that all mirrors got correct binaries.

#10 Updated by adrianSuSE 11 months ago

Do not judge from meta data files, there was no rebuild since 24th Mai. The project is locked since 26th Mai.

However, this does not mean that there can not be rsync calls, entire OBS code is written is to keep content in sync and renaming content just on the stage server is making it inconsistent. If you really want to break consistency here you need to block rsync uploads then as well.

(the proper way to do it would have been using the release mechanic, but it is way to late to set this up now).

#11 Updated by adrianSuSE 11 months ago

I added an "exit 1" at the beginning of the 15.3 product upload script now.

#12 Updated by andriinikitin 11 months ago

adrianSuSE wrote:

I added an "exit 1" at the beginning of the 15.3 product upload script now.

Ok, thx - that should fix DVD iso, but what about Live CD and etc , should 15.3:Images be disabled as well? I see they are getting rebuilt - should they be frozen?

> rsync obspublish::openqa/openSUSE:Leap:15.3:Images/images/x86_64/livecd-leap-gnome/ | grep iso$
-rw-r--r--    898,695,168 2021/06/21 21:54:34 openSUSE-Leap-15.3-GNOME-Live-x86_64-Build9.102-Media.iso

#13 Updated by adrianSuSE 11 months ago

afaik they are kept building to include maintenance updates, but not my call ...

#14 Updated by andriinikitin 11 months ago

OK I think I understand what happened.
We have two separate problems:

  1. 15.3 iso (DVD and NET) - no mirrors have the files which are available at download.o.o
  2. 15.3 Live and JeOS images are updated to new builds on mirrors, but download.o.o still shows old build.

Reasons:

  1. iso files were manually renamed when GA was released. But project's publishing wasn't disabled, so those renamed files were wiped out when OBS decided to rebuild and republish binaries.
  2. Project 15.3:Images publishes new builds periodically, I guess it is expected. But publish_distro scripts are disabled for leap153_live and leap153_appliances in publish_factory_leap script.

So proper fix will be
for 1: disable build and publishing of openSUSE:Leap:15.3 and sync content which shows download.o.o to where the mirrors sync from.

for 2: depending on the needs - the same as in 1 or enable publish_distro scripts for leap153_live and leap153_appliances

lkocman let me know if you need assistance with that.

#15 Updated by adrianSuSE 11 months ago

1) again, build is disabled since a long time, but OBS can sync at any time nevertheless, since it is his job to keep content in sync.
If you fiddle around on stage server directly you would need to block updates also.

That was always the case and the inconsistency is only guaranteed atm by breaking the sync script with the "exit 1".

#16 Updated by andriinikitin 11 months ago

adrianSuSE wrote:

...

Ah, thx for confirming this again.
(I am a bit surprised that OBS will try to keep content in sync when Publishing is disabled, but it is not the topic for this ticket)
So 1. should be fixed at sync side, I will make sure that mirrors get correct version of files.
Only 2 needs an action then

#17 Updated by andriinikitin 11 months ago

  • Status changed from New to Closed

ok, I've got confirmation that the live and JeOS images are released manually, so mirrors had correct info in /appliances
So the problem was that download.o.o was showing old files for live and JeOS images, it has been addressed.
All should be good now, closing the call

Also available in: Atom PDF