https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-10-11T10:10:01ZopenSUSE Project Management ToolopenQA Infrastructure - action #100712: Investigate what broke git checkouts on o3https://progress.opensuse.org/issues/100712?journal_id=4538812021-10-11T10:10:01Zokurzokurz@suse.com
<ul><li><strong>Assignee</strong> set to <i>okurz</i></li><li><strong>Target version</strong> set to <i>Ready</i></li></ul><p>@fvogt as I understood you already fixed the individual repos. In the past we have seen similar errors and I always tried to improve fetchneedles one step at a time. There had been no recent change in fetchneedles however. There was just <a href="https://github.com/os-autoinst/openQA/commit/986bac2a9b8a42cd8cd673f061d5407ccd893717#diff-0bb3fc4e32c66e0e4e124d1288c9e57e8e32f17d020d88fc2f085693996814f6" class="external">https://github.com/os-autoinst/openQA/commit/986bac2a9b8a42cd8cd673f061d5407ccd893717#diff-0bb3fc4e32c66e0e4e124d1288c9e57e8e32f17d020d88fc2f085693996814f6</a> in August in past months so after the incident and also it looks it can not possibly cause such error.</p>
<p>As you fixed the current situation should we still treat it as "Urgent"?</p>
<p>favogt wrote:</p>
<blockquote>
<p>All of the error logs ended up in <code>/var/spool/mail/root</code>, which probably should be archived</p>
</blockquote>
<p>What do you mean with "should be archived"?</p>
openQA Infrastructure - action #100712: Investigate what broke git checkouts on o3https://progress.opensuse.org/issues/100712?journal_id=4539652021-10-11T11:07:46Zfavogtfvogt@suse.com
<ul></ul><p>iforster has the suspicion that this might be related to/caused by the recent NetApp failure which brought down some services. I asked infra about that, maybe it fits.</p>
<p>It's a bit weird though that it would only hit <code>.git/index</code> files, and that in multiple subsequent fetch runs.</p>
<p>okurz wrote:</p>
<blockquote>
<p>@fvogt as I understood you already fixed the individual repos. In the past we have seen similar errors</p>
</blockquote>
<p>Also corrupt git files? Merge conflicts and similar probably aren't related.</p>
<blockquote>
<p>and I always tried to improve fetchneedles one step at a time. There had been no recent change in fetchneedles however. There was just <a href="https://github.com/os-autoinst/openQA/commit/986bac2a9b8a42cd8cd673f061d5407ccd893717#diff-0bb3fc4e32c66e0e4e124d1288c9e57e8e32f17d020d88fc2f085693996814f6" class="external">https://github.com/os-autoinst/openQA/commit/986bac2a9b8a42cd8cd673f061d5407ccd893717#diff-0bb3fc4e32c66e0e4e124d1288c9e57e8e32f17d020d88fc2f085693996814f6</a> in August in past months so after the incident and also it looks it can not possibly cause such error.</p>
</blockquote>
<p>Yeah, I couldn't find anything either.</p>
<blockquote>
<p>As you fixed the current situation should we still treat it as "Urgent"?</p>
</blockquote>
<p>Without knowing the cause it's not unlikely that it happens again.</p>
<blockquote>
<p>favogt wrote:</p>
<blockquote>
<p>All of the error logs ended up in <code>/var/spool/mail/root</code>, which probably should be archived</p>
</blockquote>
<p>What do you mean with "should be archived"?</p>
</blockquote>
<p>Stored elsewhere in compressed form to keep it, but make it easier to inspect for future events.</p>
openQA Infrastructure - action #100712: Investigate what broke git checkouts on o3https://progress.opensuse.org/issues/100712?journal_id=4539832021-10-11T11:22:10Zfavogtfvogt@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li></ul><p>Got an answer:</p>
<blockquote>
<p>yes, very likely related. From <a href="mailto:geekotest@ariel.suse-dmz.opensuse.org">geekotest@ariel.suse-dmz.opensuse.org</a> Sat Oct 9 22:41:09 2021 is within a few seconds of when the trouble on the host started</p>
</blockquote>
<p>So let's close this for now, and hope it doesn't happen again.</p>
openQA Infrastructure - action #100712: Investigate what broke git checkouts on o3https://progress.opensuse.org/issues/100712?journal_id=4546762021-10-12T13:41:15Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>Resolved</i></li></ul>