https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842018-12-30T08:53:46ZopenSUSE Project Management ToolopenSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1758412018-12-30T08:53:46ZAnonymous
<ul></ul><p>Dear sender</p>
<p>I'm out of office until Tuesday, 2019-01-02, and will not read my Email regulary. <br>
In urgent cases, please contact my manager, Roland Haidl <a href="mailto:rhaidl@suse.com">rhaidl@suse.com</a>.</p>
<p>You might also contact: </p>
<ul>
<li><a href="mailto:autobuild@suse.de">autobuild@suse.de</a> for all questions around Autobuild and the Build Service</li>
</ul>
<p>With kind regards<br>
Lars Vogdt</p>
<p>-- <br>
Lars Vogdt <a href="mailto:Lars.Vogdt@suse.com">Lars.Vogdt@suse.com</a> </p>
<ul>
<li>BuildOPS Team Lead -
SUSE Linux GmbH - GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer
Maxfeldstraße 5, 90409 Nuernberg, Germany - HRB 16746 (AG Nuernberg)</li>
</ul>
<blockquote>
<blockquote>
<blockquote>
<p><a href="mailto:admin@opensuse.org">admin@opensuse.org</a> 12/30/18 09:33 >>></p>
</blockquote>
</blockquote>
</blockquote>
<p>[openSUSE Tracker]<br>
Issue <a class="issue tracker-10 status-5 priority-4 priority-default closed" title="tickets: A couple of openSUSE machines run out of disk space (Closed)" href="https://progress.opensuse.org/issues/45614">#45614</a> has been reported by <a href="mailto:Lars.Vogdt@suse.com">Lars.Vogdt@suse.com</a>.</p>
<hr>
<p>tickets <a class="issue tracker-10 status-5 priority-4 priority-default closed" title="tickets: A couple of openSUSE machines run out of disk space (Closed)" href="https://progress.opensuse.org/issues/45614">#45614</a>: A couple of openSUSE machines run out of disk space<br>
<a href="https://progress.opensuse.org/issues/45614" class="external">https://progress.opensuse.org/issues/45614</a></p>
<ul>
<li>Author: <a href="mailto:Lars.Vogdt@suse.com">Lars.Vogdt@suse.com</a></li>
<li>Status: New</li>
<li>Priority: Normal</li>
<li>Assignee: </li>
<li>Category: </li>
</ul>
<a name="-Target-version"></a>
<h2 >* Target version: <a href="#-Target-version" class="wiki-anchor">¶</a></h2>
<p>Hi</p>
<p>Sorry to say, but while debugging a problem with one of the hypervisor<br>
machines, I noticed that some openSUSE machines are running out of disk<br>
space. Namely:</p>
<ul>
<li>boosters.infra.opensuse.org</li>
<li>mirrordb3.infra.opensuse.org</li>
<li>mirrordb4.infra.opensuse.org</li>
<li>narwal3.infra.opensuse.org</li>
<li>osc-collab.infra.opensuse.org</li>
</ul>
<p>Please inform the administrators of those boxes, so they can start a<br>
cleanup round.</p>
<p>Another topic:</p>
<ul>
<li>icc.infra.opensuse.org hangs </li>
<li>narwal2.infra.opensuse.org hangs in maintenance mode (see screen)</li>
</ul>
<p>Please investigate.</p>
<p>Regards<br>
Lars</p>
<p>-- <br>
You have received this notification because you have either subscribed to it, or are involved in it.<br>
To change your notification preferences, please click here: <a href="http://progress.opensuse.org/my/account" class="external">http://progress.opensuse.org/my/account</a></p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1767172019-01-07T09:31:13Ztampakraptampakrap@gmail.com
<ul><li><strong>Category</strong> set to <i>Servers hosted in NBG</i></li><li><strong>Assignee</strong> set to <i>tampakrap</i></li></ul><p>@Lars, thanks a lot for handling the hypervisor issue while everyone was on Christmas break, and also for bringing back the failed VMs, including the very important mirrordb1! Your effort is really appreciated!</p>
<p>As for the rest of the still failed VMs, I'll get to them with a bit of delay though, as I'm about to leave on a business trip for the whole week and I'll be on very limited availability.</p>
<p>A few more VMs that have been reported directly to me as broken are:</p>
<ul>
<li>aedir[1-2].i.o.o</li>
<li>lnt.i.o.o</li>
<li>the CaaSP cluster (not all of the VMs of the cluster seem to be down though, but the endpoint fails)</li>
</ul>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1767202019-01-07T09:31:25Ztampakraptampakrap@gmail.com
<ul><li><strong>Private</strong> changed from <i>Yes</i> to <i>No</i></li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1768462019-01-07T22:48:39Zcboltzsuse-beta@cboltz.de
<ul></ul><p>I have good and bad news:</p>
<p>bad: provo-mirror is also down (no idea why, I'd guess it's unrelated to the NBG hypervisor problems)</p>
<p>good 1: I manually compressed the nginx logs on narwal3 some days ago, so the disk space issue is fixed for now (interestingly, the logs were rotated, but not compressed)</p>
<p>good 2: I'm working on replacing the old narwals with some salt (both the webservers and automated git pull) and hope to have it ready in the next days, so maybe you won't need to spend too much time to fix narwal2 ;-)</p>
<p>I'll also add a checklist to the ticket (one item per server) to make sure nothing gets lost ;-)</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1768492019-01-07T22:51:36Zcboltzsuse-beta@cboltz.de
<ul><li><b>Checklist item</b> changed from to [ ] disk space: boosters, [ ] disk space: mirrordb3, [ ] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [ ] down: icc, [ ] down: narwal2, [ ] down: aedir1, [ ] down: aedir2, [ ] down: lnt, [ ] down: CaaSP cluster (endpoints fail), [ ] down: provo-mirror</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1771312019-01-08T13:28:37Zmcajmcaj@suse.com
<ul><li><b>Checklist item</b> changed from [ ] disk space: boosters, [ ] disk space: mirrordb3, [ ] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [ ] down: icc, [ ] down: narwal2, [ ] down: aedir1, [ ] down: aedir2, [ ] down: lnt, [ ] down: CaaSP cluster (endpoints fail), [ ] down: provo-mirror to [ ] disk space: boosters, [ ] disk space: mirrordb3, [ ] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [ ] down: icc, [ ] down: narwal2, [ ] down: aedir1, [ ] down: aedir2, [x] down: lnt, [ ] down: CaaSP cluster (endpoints fail), [ ] down: provo-mirror</li></ul><p>FYI I checked the status of the machine lnt.infra.opensuse.org aka lnt.opensuse.org.</p>
<p>The machine was not responding on ping. I found only one message on the serial console output:</p>
<p>[16776824.048003] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]</p>
<p>I was not able to log there via virt-manager. The machine did not react on soft reboot so I had to do <br>
the force reboot. </p>
<p>After the force reboot its seem to be up and running. Also the web <a href="https://lnt.opensuse.org/" class="external">https://lnt.opensuse.org/</a> is working.<br>
But admin of the machine should check logs of the machine.</p>
<p>Martin</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1772812019-01-08T14:05:21Zmcajmcaj@suse.com
<ul></ul><p>The VM machine icc is broken and reboot does not help.</p>
<p>The is a message from kernel:<br>
Probing EDD (edd=off to disable)... ok</p>
<p>and then this message :</p>
<p>PANIC early exception 0d rip 10:ffffffff810321f5 error 0 cr2 0</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1774072019-01-08T17:42:57Zcboltzsuse-beta@cboltz.de
<ul></ul><p>mcaj wrote:</p>
<blockquote>
<p>The VM machine icc is broken and reboot does not help.</p>
<p>The is a message from kernel: [...]<br>
PANIC early exception 0d rip 10:ffffffff810321f5 error 0 cr2 0</p>
</blockquote>
<p>Wild guess: try booting the previous kernel</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1774192019-01-08T22:19:25Zcboltzsuse-beta@cboltz.de
<ul><li><b>Checklist item</b> changed from [ ] disk space: boosters, [ ] disk space: mirrordb3, [ ] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [ ] down: icc, [ ] down: narwal2, [ ] down: aedir1, [ ] down: aedir2, [x] down: lnt, [ ] down: CaaSP cluster (endpoints fail), [ ] down: provo-mirror to [ ] disk space: boosters, [ ] disk space: mirrordb3, [ ] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [ ] down: icc, [ ] down: narwal2, [ ] down: aedir1, [ ] down: aedir2, [x] down: lnt, [ ] down: CaaSP cluster (endpoints fail), [ ] down: 101.opensuse.org (CaaSP?), [ ] down: provo-mirror</li></ul><p>101.opensuse.org shows "404 Not Found: Requested route ('101.cf.infra.opensuse.org') does not exist.", added to the checklist</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1787872019-01-11T16:54:17Zcboltzsuse-beta@cboltz.de
<ul><li><b>Checklist item</b> changed from [ ] disk space: boosters, [ ] disk space: mirrordb3, [ ] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [ ] down: icc, [ ] down: narwal2, [ ] down: aedir1, [ ] down: aedir2, [x] down: lnt, [ ] down: CaaSP cluster (endpoints fail), [ ] down: 101.opensuse.org (CaaSP?), [ ] down: provo-mirror to [ ] disk space: boosters, [ ] disk space: mirrordb3, [ ] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [ ] down: icc, [ ] down: narwal2, [ ] down: aedir1, [ ] down: aedir2, [x] down: lnt, [ ] down: CaaSP cluster (endpoints fail), [ ] down: 101.opensuse.org (CaaSP?), [x] down: provo-mirror</li></ul><p>provo-mirror is back since about 17 hours - and we instantly got ticket <a class="issue tracker-10 status-5 priority-4 priority-default closed" title="tickets: provo-mirror.opensuse.org is outdated (Closed)" href="https://progress.opensuse.org/issues/46031">#46031</a> because it's outdated ;-)</p>
<p>Thanks to whoever brought provo-mirror back!</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1787902019-01-11T17:23:54ZAnonymous
<ul></ul><p>Am Fri, 11 Jan 2019 16:54:18 +0000<br>
schrieb <a href="mailto:admin@opensuse.org">admin@opensuse.org</a>:</p>
<blockquote>
<p>provo-mirror is back since about 17 hours - and we instantly got<br>
ticket <a class="issue tracker-10 status-5 priority-4 priority-default closed" title="tickets: provo-mirror.opensuse.org is outdated (Closed)" href="https://progress.opensuse.org/issues/46031">#46031</a> because it's outdated ;-)</p>
<p>Thanks to whoever brought provo-mirror back!</p>
</blockquote>
<p>FYI: provo-mirror had "disk full". Luckily we found someone with big<br>
pockets at SUSE who sponsored some more space (30TB).</p>
<p>The machine is and will provide outdated packages for the weekend<br>
12./13.Jan as we decided to stop updating with latest builds but<br>
instead speeding up the sync of the underlying lvm move process. </p>
<p>provo-mirror should be back on track (and hopefully stay online and<br>
up-to date for a longer time) early next week. Until than, it might be<br>
a good idea to rely on download.opensuse.org to get the latest<br>
packages. For installation media and some (not updated) packages or<br>
repositories, the packages on provo-mirror should be good enough<br>
(that's the reason why we leave it online). Thankfully MirrorBrain<br>
behind download.opensuse.org knows which packages or ISO images can be<br>
used and which not - and will redirect you to other mirrors in case the<br>
files on provo-mirror are outdated.</p>
<p>I hope this explains the situation.</p>
<p>With kind regards,<br>
Lars</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1787992019-01-11T20:04:34Zcboltzsuse-beta@cboltz.de
<ul></ul><p>That's the best reason I ever heard for making a server read-only :-)</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1793842019-01-15T11:16:11Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] down: CaaSP cluster (endpoints fail)</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1793872019-01-15T11:17:31Ztampakraptampakrap@gmail.com
<ul></ul><p>all CaaSP nodes are back up again. Also, the NFS server that k8s uses as storage was also down. I brought it up but it still didn't catch up. Thus cloud foundry and the websites on top of it are down atm</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1794142019-01-15T12:20:36Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] disk space: mirrordb3</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1794172019-01-15T12:20:37Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] disk space: mirrordb4</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1794202019-01-15T12:21:11Ztampakraptampakrap@gmail.com
<ul></ul><p>I marked mirrordb3/4 as done because they are not actually used any more, and they are pending destruction. I'm waiting for darix's ok first</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1802062019-01-17T14:43:41Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] down: 101.opensuse.org (CaaSP?)</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1802092019-01-17T14:45:10Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] down: aedir1</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1802122019-01-17T14:45:11Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] down: aedir2</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1802332019-01-17T15:00:15Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] down: icc</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1806592019-01-19T20:32:06Zcboltzsuse-beta@cboltz.de
<ul><li><b>Checklist item</b> changed from [ ] disk space: boosters, [x] disk space: mirrordb3, [x] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [x] down: icc, [ ] down: narwal2, [x] down: aedir1, [x] down: aedir2, [x] down: lnt, [x] down: CaaSP cluster (endpoints fail), [x] down: 101.opensuse.org (CaaSP?), [x] down: provo-mirror to [ ] disk space: boosters, [x] disk space: mirrordb3, [x] disk space: mirrordb4, [x] disk space: narwal3, [ ] disk space: osc-collab, [ ] down: icc, [ ] down: narwal2, [x] down: aedir1, [x] down: aedir2, [x] down: lnt, [x] down: CaaSP cluster (endpoints fail), [x] down: 101.opensuse.org (CaaSP?), [x] down: provo-mirror</li></ul><p>icc.o.o still shows the 503 maintenance page :-(</p>
<p>I can ping the VM, so maybe "only" the service is down.</p>
openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1875922019-02-13T11:28:45Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] down: narwal2</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1875952019-02-13T11:28:49Ztampakraptampakrap@gmail.com
<ul><li><b>Checklist item</b> changed from to [x] disk space: boosters</li></ul> openSUSE admin - tickets #45614: A couple of openSUSE machines run out of disk spacehttps://progress.opensuse.org/issues/45614?journal_id=1981942019-03-12T13:01:51Ztampakraptampakrap@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li></ul><p>closing this one as icc and osc-collab have dedicated maintainers that are aware of the issues already. Anyone feel free to file separate tickets for those</p>