openSUSE admin

I think the the problem here is that the mirrors got new version, but scanners couldn't update the info for some reasons.

I am not sure what to do exactly and how to troubleshoot such problem, but I decided that at least scanning manually the problem servers should fix 404, so I did:

/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso widehat.opensuse.org
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso widehat.opensuse.org

The last command was just hanging, so I've looked in mirrordb2 and had to kill some stalled queries and command succeeded.
Should be fixed now, at least for most of countries, I will try to make sure that all mirrors from mirrorlist are rescanned for these locations

pjessen wrote:

We are certain this is in no way related to the linking of ftp-stage to ftp ? Scanning ought to be running continuously from olaf. It seems very odd if there is a problem with scanning that only affects Leap 15.3.

I think the problem is both with scanning and publishing of 15.3 iso, but it is hard to tell which exactly

Now I think I did understand the problem wrong from the start. Leap 15.3 was released already, so no new .iso must be published anymore, correct?
So the problem is that .iso got updated on mirrors, when it shouldn't happen anymore, correct?

It seems to work now for me

I could get e.g. without any issues (same for aarch64 and intel)
https://download.opensuse.org/distribution/leap/15.3/iso/openSUSE-Leap-15.3-DVD-ppc64le.iso

Same for links in http://get.opensuse.org/leap/#download

Lubos

andriinikitin wrote:

Now I think I did understand the problem wrong from the start. Leap 15.3 was released already, so no new .iso must be published anymore, correct?

That is correct. Looking at download.o.o/distribution/opensuse/leap/15.3/iso, they are all dated 26 May 13:36. On my mirror (Hostsuisse), the ISO images are dated 24 May 13:46 to 14:25. This corresponds to contents of ftp-stage whereas download.o.o is showing /srv/ftp.

So the problem is that .iso got updated on mirrors, when it shouldn't happen anymore, correct?

I don't know where mirrors would get new isos from - mirrors rsync from ftp-stage.
Looking at ftp.gwdg.de, the isos are dated May 24.

pjessen wrote:

Looking at ftp.gwdg.de, the isos are dated May 24.

Ah, then the problem is that most of mirrors now have "-Media" in their name, e.g. gwdg.de has:
openSUSE-Leap-15.3-DVD-x86_64-Media.iso
While download.o.o has:
openSUSE-Leap-15.3-DVD-x86_64.iso

And the the mirrors were updated few days ago recently, because mirrorbrain did have info about correct filename on the mirrors.

andriinikitin wrote:

And the the mirrors were updated few days ago recently, because mirrorbrain did have info about correct filename on the mirrors.

and in appliances the files are from June with Build9.96 http://ftp.gwdg.de/pub/opensuse/distribution/leap/15.3/appliances/iso/
while download.o.o has Build9.88 , so these were new releases of 15.3 - not sure what went wrong

Do not judge from meta data files, there was no rebuild since 24th Mai. The project is locked since 26th Mai.

However, this does not mean that there can not be rsync calls, entire OBS code is written is to keep content in sync and renaming content just on the stage server is making it inconsistent. If you really want to break consistency here you need to block rsync uploads then as well.

(the proper way to do it would have been using the release mechanic, but it is way to late to set this up now).

I added an "exit 1" at the beginning of the 15.3 product upload script now.

adrianSuSE wrote:

I added an "exit 1" at the beginning of the 15.3 product upload script now.

Ok, thx - that should fix DVD iso, but what about Live CD and etc , should 15.3:Images be disabled as well? I see they are getting rebuilt - should they be frozen?

> rsync obspublish::openqa/openSUSE:Leap:15.3:Images/images/x86_64/livecd-leap-gnome/ | grep iso$
-rw-r--r--    898,695,168 2021/06/21 21:54:34 openSUSE-Leap-15.3-GNOME-Live-x86_64-Build9.102-Media.iso

afaik they are kept building to include maintenance updates, but not my call ...

OK I think I understand what happened.
We have two separate problems:

1. 15.3 iso (DVD and NET) - no mirrors have the files which are available at download.o.o
2. 15.3 Live and JeOS images are updated to new builds on mirrors, but download.o.o still shows old build.

Reasons:

1. iso files were manually renamed when GA was released. But project's publishing wasn't disabled, so those renamed files were wiped out when OBS decided to rebuild and republish binaries.
2. Project 15.3:Images publishes new builds periodically, I guess it is expected. But publish_distro scripts are disabled for leap153_live and leap153_appliances in publish_factory_leap script.

So proper fix will be
for 1: disable build and publishing of openSUSE:Leap:15.3 and sync content which shows download.o.o to where the mirrors sync from.

for 2: depending on the needs - the same as in 1 or enable publish_distro scripts for leap153_live and leap153_appliances

@lkocman let me know if you need assistance with that.

1) again, build is disabled since a long time, but OBS can sync at any time nevertheless, since it is his job to keep content in sync.
If you fiddle around on stage server directly you would need to block updates also.

That was always the case and the inconsistency is only guaranteed atm by breaking the sync script with the "exit 1".

adrianSuSE wrote:

...

Ah, thx for confirming this again.
(I am a bit surprised that OBS will try to keep content in sync when Publishing is disabled, but it is not the topic for this ticket)
So 1. should be fixed at sync side, I will make sure that mirrors get correct version of files.
Only 2 needs an action then