https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-06-19T10:03:49ZopenSUSE Project Management ToolopenSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4177722021-06-19T10:03:49Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>I think the the problem here is that the mirrors got new version, but scanners couldn't update the info for some reasons.</p>
<p>I am not sure what to do exactly and how to troubleshoot such problem, but I decided that at least scanning manually the problem servers should fix 404, so I did:</p>
<pre><code>/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso widehat.opensuse.org
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso widehat.opensuse.org
</code></pre>
<p>The last command was just hanging, so I've looked in mirrordb2 and had to kill some stalled queries and command succeeded.<br>
Should be fixed now, at least for most of countries, I will try to make sure that all mirrors from mirrorlist are rescanned for these locations</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4177752021-06-19T10:40:43Zpjessenper@computer.org
<ul><li><strong>Tracker</strong> changed from <i>communication</i> to <i>tickets</i></li><li><strong>Subject</strong> changed from <i>All Leap downloads are broken</i> to <i>All Leap 15.3 downloads are broken</i></li><li><strong>Private</strong> changed from <i>Yes</i> to <i>No</i></li></ul><p>andriinikitin wrote:</p>
<blockquote>
<p>I think the the problem here is that the mirrors got new version, but scanners couldn't update the info for some reasons.</p>
<p>I am not sure what to do exactly and how to troubleshoot such problem, but I decided that at least scanning manually the problem servers should fix 404, so I did:</p>
<pre><code>/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso widehat.opensuse.org
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso ftp.gwdg.de
/usr/bin/perl -w /usr/bin/scanner -b mirrordb2 -d /distribution/leap/15.3/appliances/iso widehat.opensuse.org
</code></pre>
<p>The last command was just hanging, so I've looked in mirrordb2 and had to kill some stalled queries and command succeeded.<br>
Should be fixed now, at least for most of countries, I will try to make sure that all mirrors from mirrorlist are rescanned for these locations</p>
</blockquote>
<p>We are certain this is in no way related to the linking of ftp-stage to ftp ? Scanning ought to be running continuously from olaf. It seems very odd if there is a problem with scanning that only affects Leap 15.3.</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4177782021-06-19T11:05:04Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>pjessen wrote:</p>
<blockquote>
<p>We are certain this is in no way related to the linking of ftp-stage to ftp ? Scanning ought to be running continuously from olaf. It seems very odd if there is a problem with scanning that only affects Leap 15.3.</p>
</blockquote>
<p>I think the problem is both with scanning and publishing of 15.3 iso, but it is hard to tell which exactly</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4177932021-06-19T13:57:44Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>Now I think I did understand the problem wrong from the start. Leap 15.3 was released already, so no new .iso must be published anymore, correct?<br>
So the problem is that .iso got updated on mirrors, when it shouldn't happen anymore, correct?</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4177962021-06-19T14:11:04Zlkocmanlubos.kocman@suse.com
<ul></ul><p>It seems to work now for me</p>
<p>I could get e.g. without any issues (same for aarch64 and intel)<br>
<a href="https://download.opensuse.org/distribution/leap/15.3/iso/openSUSE-Leap-15.3-DVD-ppc64le.iso" class="external">https://download.opensuse.org/distribution/leap/15.3/iso/openSUSE-Leap-15.3-DVD-ppc64le.iso</a></p>
<p>Same for links in <a href="http://get.opensuse.org/leap/#download" class="external">http://get.opensuse.org/leap/#download</a></p>
<p>Lubos</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4177992021-06-19T15:47:40Zpjessenper@computer.org
<ul></ul><p>andriinikitin wrote:</p>
<blockquote>
<p>Now I think I did understand the problem wrong from the start. Leap 15.3 was released already, so no new .iso must be published anymore, correct?</p>
</blockquote>
<p>That is correct. Looking at download.o.o/distribution/opensuse/leap/15.3/iso, they are all dated 26 May 13:36. On my mirror (Hostsuisse), the ISO images are dated 24 May 13:46 to 14:25. This corresponds to contents of ftp-stage whereas download.o.o is showing /srv/ftp. </p>
<blockquote>
<p>So the problem is that .iso got updated on mirrors, when it shouldn't happen anymore, correct?</p>
</blockquote>
<p>I don't know where mirrors would get new isos from - mirrors rsync from ftp-stage. <br>
Looking at ftp.gwdg.de, the isos are dated May 24. </p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4178112021-06-19T16:57:58Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>pjessen wrote:</p>
<blockquote>
<p>Looking at ftp.gwdg.de, the isos are dated May 24.</p>
</blockquote>
<p>Ah, then the problem is that most of mirrors now have "-Media" in their name, e.g. gwdg.de has:<br>
openSUSE-Leap-15.3-DVD-x86_64-Media.iso <br>
While download.o.o has:<br>
openSUSE-Leap-15.3-DVD-x86_64.iso</p>
<p>And the the mirrors were updated few days ago recently, because mirrorbrain did have info about correct filename on the mirrors.</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4178142021-06-19T17:09:01Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>andriinikitin wrote:</p>
<blockquote>
<p>And the the mirrors were updated few days ago recently, because mirrorbrain did have info about correct filename on the mirrors.</p>
</blockquote>
<p>and in appliances the files are from June with Build9.96 <a href="http://ftp.gwdg.de/pub/opensuse/distribution/leap/15.3/appliances/iso/" class="external">http://ftp.gwdg.de/pub/opensuse/distribution/leap/15.3/appliances/iso/</a><br>
while download.o.o has Build9.88 , so these were new releases of 15.3 - not sure what went wrong</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4188282021-06-22T07:23:23Zandriinikitinandrii.nikitin@suse.com
<ul><li><strong>Assignee</strong> set to <i>lkocman</i></li></ul><p>So it looks the reason for the problem is that Leap 15.3 was rebuilt in OBS (probably some of dependencies changed) :</p>
<pre><code>> rsync obspublish::openqa/openSUSE:Leap:15.3/standard/x86_64/ | grep 2021/06
drwxr-xr-x 8,192 2021/06/18 09:18:50 .
-rw-r--r-- 67,477 2021/06/18 09:15:10 :full.metacache
-rw-r--r-- 555,409 2021/06/18 09:08:06 :full.solv
-rw-r--r-- 228,828 2021/06/18 09:08:07 :full.useforbuild
-rw-r--r-- 1,437,129 2021/06/18 09:18:50 :full.xcache
-rw-r--r-- 371,165 2021/06/18 09:15:10 :packstatus
-rw-r--r-- 123 2021/06/18 09:15:10 :schedulerstate
</code></pre>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/32123">@lkocman</a> could you please completely disable building and publishing of Leap-15.3 project in OBS <a href="https://build.opensuse.org/repositories/openSUSE:Leap:15.3" class="external">https://build.opensuse.org/repositories/openSUSE:Leap:15.3</a> ? <br>
And I guess the same for Leap-15.3:Images project.<br>
Then assign the ticket back to me and I will make sure that all mirrors got correct binaries.</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4188372021-06-22T07:37:45ZadrianSuSEadrian.schroeter@suse.com
<ul></ul><p>Do not judge from meta data files, there was no rebuild since 24th Mai. The project is locked since 26th Mai.</p>
<p>However, this does not mean that there can not be rsync calls, entire OBS code is written is to keep content in sync and renaming content just on the stage server is making it inconsistent. If you really want to break consistency here you need to block rsync uploads then as well.</p>
<p>(the proper way to do it would have been using the release mechanic, but it is way to late to set this up now).</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4188462021-06-22T07:41:31ZadrianSuSEadrian.schroeter@suse.com
<ul></ul><p>I added an "exit 1" at the beginning of the 15.3 product upload script now.</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4188522021-06-22T07:48:55Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>adrianSuSE wrote:</p>
<blockquote>
<p>I added an "exit 1" at the beginning of the 15.3 product upload script now.</p>
</blockquote>
<p>Ok, thx - that should fix DVD iso, but what about Live CD and etc , should 15.3:Images be disabled as well? I see they are getting rebuilt - should they be frozen?</p>
<pre><code>> rsync obspublish::openqa/openSUSE:Leap:15.3:Images/images/x86_64/livecd-leap-gnome/ | grep iso$
-rw-r--r-- 898,695,168 2021/06/21 21:54:34 openSUSE-Leap-15.3-GNOME-Live-x86_64-Build9.102-Media.iso
</code></pre> openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4188552021-06-22T07:50:22ZadrianSuSEadrian.schroeter@suse.com
<ul></ul><p>afaik they are kept building to include maintenance updates, but not my call ...</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4189182021-06-22T09:44:50Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>OK I think I understand what happened.<br>
We have two separate problems:</p>
<ol>
<li>15.3 iso (DVD and NET) - no mirrors have the files which are available at download.o.o</li>
<li>15.3 Live and JeOS images are updated to new builds on mirrors, but download.o.o still shows old build.</li>
</ol>
<p>Reasons:</p>
<ol>
<li>iso files were manually renamed when GA was released. But project's publishing wasn't disabled, so those renamed files were wiped out when OBS decided to rebuild and republish binaries.</li>
<li>Project 15.3:Images publishes new builds periodically, I guess it is expected. But <code>publish_distro</code> scripts are disabled for leap153_live and leap153_appliances in publish_factory_leap script. </li>
</ol>
<p>So proper fix will be <br>
for 1: disable build and publishing of openSUSE:Leap:15.3 and sync content which shows download.o.o to where the mirrors sync from.</p>
<p>for 2: depending on the needs - the same as in 1 or enable publish_distro scripts for leap153_live and leap153_appliances</p>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/32123">@lkocman</a> let me know if you need assistance with that.</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4189242021-06-22T09:52:40ZadrianSuSEadrian.schroeter@suse.com
<ul></ul><p>1) again, build is disabled since a long time, but OBS can sync at any time nevertheless, since it is his job to keep content in sync.<br>
If you fiddle around on stage server directly you would need to block updates also.</p>
<p>That was always the case and the inconsistency is only guaranteed atm by breaking the sync script with the "exit 1".</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4189362021-06-22T10:20:44Zandriinikitinandrii.nikitin@suse.com
<ul></ul><p>adrianSuSE wrote:</p>
<blockquote>
<p>...</p>
</blockquote>
<p>Ah, thx for confirming this again. <br>
(I am a bit surprised that OBS will try to keep content in sync when Publishing is disabled, but it is not the topic for this ticket)<br>
So 1. should be fixed at sync side, I will make sure that mirrors get correct version of files.<br>
Only 2 needs an action then</p>
openSUSE admin - tickets #94273: All Leap 15.3 downloads are brokenhttps://progress.opensuse.org/issues/94273?journal_id=4196022021-06-23T11:25:14Zandriinikitinandrii.nikitin@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li></ul><p>ok, I've got confirmation that the live and JeOS images are released manually, so mirrors had correct info in /appliances <br>
So the problem was that download.o.o was showing old files for live and JeOS images, it has been addressed.<br>
All should be good now, closing the call</p>