https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842022-09-21T08:16:44ZopenSUSE Project Management ToolopenQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5555322022-09-21T08:16:44Zwaynechen55wchen@suse.com
<ul><li><strong>Project</strong> changed from <i>openQA Project</i> to <i>openQA Infrastructure</i></li></ul> openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5556102022-09-21T10:33:16Zokurzokurz@suse.com
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Urgent</i></li><li><strong>Target version</strong> set to <i>Ready</i></li></ul> openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5556132022-09-21T10:36:52Zokurzokurz@suse.com
<ul></ul><p>I manually fixed that now.</p>
<pre><code>openqa:/home/okurz # sudo -u geekotest /usr/share/openqa/script/fetchneedles
fatal: Unable to create '/var/lib/openqa/share/tests/opensuse/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
error: could not detach HEAD
Use force=1 to discard uncommitted changes before rebasing
openqa:/home/okurz # sudo -u geekotest bash -ex /usr/share/openqa/script/fetchneedles
+ : geekotest
+ : www
+ : openSUSE
+ : opensuse
+ : https://github.com/os-autoinst/os-autoinst-distri-opensuse.git
+ : master
+ : openqa@
+ : 'openQA web UI'
+ : opensuse
+ : 0
+ : 1
+ : https://github.com/os-autoinst/os-autoinst-needles-opensuse.git
+ : master
+ : 0
+ : 0
+ '[' '' = -h ']'
+ '[' '' = --help ']'
+ dir=/var/lib/openqa/share/tests
+ '[' -w / ']'
+ '[' 0 = 1 ']'
+ target=/var/lib/openqa/share/tests/opensuse
+ mkdir -p /var/lib/openqa/share/tests/opensuse
+ cd /var/lib/openqa/share/tests/opensuse
+ '[' '!' -d .git ']'
+ do_fetch /var/lib/openqa/share/tests/opensuse
+ target=/var/lib/openqa/share/tests/opensuse
+ git_update master
+ branch=master
+ git gc --auto --quiet
+ git fetch -q origin
+ '[' 0 = 1 ']'
+ git rebase -q origin/master
fatal: Unable to create '/var/lib/openqa/share/tests/opensuse/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
error: could not detach HEAD
+ fail 'Use force=1 to discard uncommitted changes before rebasing'
+ echo 'Use force=1 to discard uncommitted changes before rebasing'
Use force=1 to discard uncommitted changes before rebasing
+ exit 1
openqa:/home/okurz # cat /var/lib/openqa/share/tests/opensuse/.git/index.lock
openqa:/home/okurz # ls -ltra /var/lib/openqa/share/tests/opensuse/.git/index.lock
-rw-r--r-- 1 geekotest nogroup 0 Sep 20 02:34 /var/lib/openqa/share/tests/opensuse/.git/index.lock
openqa:/home/okurz # rm /var/lib/openqa/share/tests/opensuse/.git/index.lock
openqa:/home/okurz # sudo -u geekotest bash -ex /usr/share/openqa/script/fetchneedles
+ : geekotest
+ : www
+ : openSUSE
+ : opensuse
+ : https://github.com/os-autoinst/os-autoinst-distri-opensuse.git
+ : master
+ : openqa@
+ : 'openQA web UI'
+ : opensuse
+ : 0
+ : 1
+ : https://github.com/os-autoinst/os-autoinst-needles-opensuse.git
+ : master
+ : 0
+ : 0
+ '[' '' = -h ']'
+ '[' '' = --help ']'
+ dir=/var/lib/openqa/share/tests
+ '[' -w / ']'
+ '[' 0 = 1 ']'
+ target=/var/lib/openqa/share/tests/opensuse
+ mkdir -p /var/lib/openqa/share/tests/opensuse
+ cd /var/lib/openqa/share/tests/opensuse
+ '[' '!' -d .git ']'
+ do_fetch /var/lib/openqa/share/tests/opensuse
+ target=/var/lib/openqa/share/tests/opensuse
+ git_update master
+ branch=master
+ git gc --auto --quiet
+ git fetch -q origin
+ '[' 0 = 1 ']'
+ git rebase -q origin/master
+ '[' 1 = 1 ']'
+ test -d products
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/caasp/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/casp/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/leap-micro/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/microos/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/opensuse/needles ']'
+ cd /var/lib/openqa/share/tests/opensuse/products/opensuse/needles
+ git_update_needles
+ git_update master
+ branch=master
+ git gc --auto --quiet
+ git fetch -q origin
+ '[' 0 = 1 ']'
+ git rebase -q origin/master
fatal: Unable to create '/var/lib/openqa/share/tests/opensuse/products/opensuse/needles/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
error: could not detach HEAD
+ fail 'Use force=1 to discard uncommitted changes before rebasing'
+ echo 'Use force=1 to discard uncommitted changes before rebasing'
Use force=1 to discard uncommitted changes before rebasing
+ exit 1
openqa:/home/okurz # flock ^C
openqa:/home/okurz # lsof /var/lib/openqa/share/tests/opensuse/products/opensuse/needles/.git/index.lock
openqa:/home/okurz # echo $?
1
openqa:/home/okurz # rm /var/lib/openqa/share/tests/opensuse/products/opensuse/needles/.git/index.lock
openqa:/home/okurz # sudo -u geekotest bash -ex /usr/share/openqa/script/fetchneedles
+ : geekotest
+ : www
+ : openSUSE
+ : opensuse
+ : https://github.com/os-autoinst/os-autoinst-distri-opensuse.git
+ : master
+ : openqa@
+ : 'openQA web UI'
+ : opensuse
+ : 0
+ : 1
+ : https://github.com/os-autoinst/os-autoinst-needles-opensuse.git
+ : master
+ : 0
+ : 0
+ '[' '' = -h ']'
+ '[' '' = --help ']'
+ dir=/var/lib/openqa/share/tests
+ '[' -w / ']'
+ '[' 0 = 1 ']'
+ target=/var/lib/openqa/share/tests/opensuse
+ mkdir -p /var/lib/openqa/share/tests/opensuse
+ cd /var/lib/openqa/share/tests/opensuse
+ '[' '!' -d .git ']'
+ do_fetch /var/lib/openqa/share/tests/opensuse
+ target=/var/lib/openqa/share/tests/opensuse
+ git_update master
+ branch=master
+ git gc --auto --quiet
+ git fetch -q origin
+ '[' 0 = 1 ']'
+ git rebase -q origin/master
+ '[' 1 = 1 ']'
+ test -d products
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/caasp/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/casp/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/leap-micro/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/microos/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/opensuse/needles ']'
+ cd /var/lib/openqa/share/tests/opensuse/products/opensuse/needles
+ git_update_needles
+ git_update master
+ branch=master
+ git gc --auto --quiet
+ git fetch -q origin
+ '[' 0 = 1 ']'
+ git rebase -q origin/master
++ git rev-parse --abbrev-ref --symbolic-full-name HEAD
+ '[' master = HEAD ']'
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/sle-micro/needles ']'
+ continue
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/sle/needles ']'
+ cd /var/lib/openqa/share/tests/opensuse/products/sle/needles
+ git_update_needles
+ git_update master
+ branch=master
+ git gc --auto --quiet
+ git fetch -q origin
+ '[' 0 = 1 ']'
+ git rebase -q origin/master
++ git rev-parse --abbrev-ref --symbolic-full-name HEAD
+ '[' master = HEAD ']'
+ for nd in products/*/needles
+ '[' -h /var/lib/openqa/share/tests/opensuse/products/windows/needles ']'
+ cd /var/lib/openqa/share/tests/opensuse/products/windows/needles
+ git_update_needles
+ git_update master
+ branch=master
+ git gc --auto --quiet
+ git fetch -q origin
+ '[' 0 = 1 ']'
+ git rebase -q origin/master
++ git rev-parse --abbrev-ref --symbolic-full-name HEAD
+ '[' master = HEAD ']'
</code></pre>
<p>I think we should extend fetchneedles to call "lsof" on the lock file and remove the file if not used by any process (exit code 1 of lsof). Or can git somehow handle that better itself?</p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5556192022-09-21T10:37:38Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/116722">action #116722</a>: openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics size:M</i> added</li></ul> openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5556312022-09-21T10:57:04Zwaynechen55wchen@suse.com
<ul></ul><p>Still failed and same error. Will try again tomorrow.</p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5556732022-09-21T12:08:16Zmkittlermarius.kittler@suse.com
<ul><li><strong>Assignee</strong> set to <i>mkittler</i></li></ul> openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5556822022-09-21T12:15:09Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li></ul><p>Please try whether it works again. I've just removed the lock file which is likely a leftover of the unclean exit due to yet another kernel panic (see <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics si... (Resolved)" href="https://progress.opensuse.org/issues/116722">#116722</a>).</p>
<p>There are now a few untracked changes left in the repo. I suppose they don't hurt anyone so I'll leave them there in case somebody wants to recover the files from <code>openqa.suse.de:/var/lib/openqa/share/tests/sle/products/sle/needles</code>.</p>
<p>I suppose it'll not be a good idea to re-trigger failed Minion jobs (<a href="https://openqa.suse.de/minion/jobs?state=failed&task=save_needle" class="external">https://openqa.suse.de/minion/jobs?state=failed&task=save_needle</a>) because the changes might have already been otherwise committed and some of the failures might be duplicates (from users retrying). So I'll just delete them later.</p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5557422022-09-21T12:25:06Zokurzokurz@suse.com
<ul></ul><p>mkittler wrote:</p>
<blockquote>
<p>Please try whether it works again. I've just removed the lock file which is likely a leftover of the unclean exit due to yet another kernel panic (see <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: openqa.suse.de is not reachable 2022-09-18, no ping response, postgreSQL OOM and kernel panics si... (Resolved)" href="https://progress.opensuse.org/issues/116722">#116722</a>).</p>
</blockquote>
<p>I have already done that some hours ago, see <a class="issue tracker-4 status-3 priority-6 priority-high2 closed" title="action: [openQA][needle] Can not commit new needle for test suite on openqa.suse.de (Resolved)" href="https://progress.opensuse.org/issues/116911#note-3">#116911#note-3</a>. Were there problems right now again?</p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5562402022-09-22T12:21:29Zmkittlermarius.kittler@suse.com
<ul></ul><p>Not sure what you did but the error <code>fatal: Unable to create '/var/lib/openqa/share/tests/opensuse/.git/index.lock': File exists.</code> persisted (see also the comment of <a class="user active user-mention" href="https://progress.opensuse.org/users/29776">@waynechen55</a>).</p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5563992022-09-22T17:15:54Zokurzokurz@suse.com
<ul></ul><p>mkittler wrote:</p>
<blockquote>
<p>Not sure what you did but the error <code>fatal: Unable to create '/var/lib/openqa/share/tests/opensuse/.git/index.lock': File exists.</code> persisted (see also the comment of <a class="user active user-mention" href="https://progress.opensuse.org/users/29776">@waynechen55</a>).</p>
</blockquote>
<p>Well, what I did is what is written in <a class="issue tracker-4 status-3 priority-6 priority-high2 closed" title="action: [openQA][needle] Can not commit new needle for test suite on openqa.suse.de (Resolved)" href="https://progress.opensuse.org/issues/116911#note-3">#116911#note-3</a> which shows that after handling the lock files fetchneedles completed without any errors.</p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5597712022-10-04T11:35:31Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>So it must've either broken again. However, it hasn't happened yet another time (<a href="https://openqa.suse.de/minion/jobs?state=failed&task=save_needle" class="external">https://openqa.suse.de/minion/jobs?state=failed&task=save_needle</a> is empty at the time of writing this comment) so I suppose the issue can be resolved. (And the crashes of OSD that were causing the issue in the first place should now at least be worked around correctly.)</p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5610162022-10-07T09:19:07Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Feedback</i></li></ul><p>I still think we should extend fetchneedles to call "lsof" on the lock file and remove the file if not used by any process (exit code 1 of lsof). Or can git somehow handle that better itself?</p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5611002022-10-07T12:48:15Zmkittlermarius.kittler@suse.com
<ul></ul><p>PR for that: <a href="https://github.com/os-autoinst/openQA/pull/4835" class="external">https://github.com/os-autoinst/openQA/pull/4835</a></p>
openQA Infrastructure - action #116911: [openQA][needle] Can not commit new needle for test suite on openqa.suse.dehttps://progress.opensuse.org/issues/116911?journal_id=5614872022-10-10T10:12:18Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>The changes have been deployed on OSD and tests/needles are still synced. So the changes at least did not break the normal operation of the script.</p>
<p>I've also added a stale lock file and it seems to be removed correctly:</p>
<pre><code>sudo -u geekotest bash -ex /usr/share/openqa/script/fetchneedles
…
+ lock_path=.git/index.lock
+ '[' -e .git/index.lock ']'
+ fuser --silent .git/index.lock
+ echo 'removing stale lock /var/lib/openqa/share/tests/opensuse/.git/index.lock'
removing stale lock /var/lib/openqa/share/tests/opensuse/.git/index.lock
+ rm .git/index.lock
+ git gc --auto --quiet
+ git fetch -q origin
…
</code></pre>