https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-09-24T11:11:17ZopenSUSE Project Management ToolopenQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4491152021-09-24T11:11:17Zokurzokurz@suse.com
<ul><li><strong>Copied from</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" href="/issues/75241">action #75241</a>: Upgrade o3 webUI host to openSUSE Leap 15.2</i> added</li></ul> openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4492022021-09-24T11:31:02Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>Upgrade o3 webUI host to openSUSE Leap 15.2</i> to <i>Upgrade o3 webUI host to openSUSE Leap 15.3</i></li><li><strong>Assignee</strong> deleted (<del><i>mkittler</i></del>)</li><li><strong>Priority</strong> changed from <i>High</i> to <i>Normal</i></li><li><strong>Start date</strong> deleted (<del><i>2020-10-24</i></del>)</li></ul> openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4511462021-09-30T09:48:27Zlivdywanliv.dywan@suse.com
<ul><li><strong>Subject</strong> changed from <i>Upgrade o3 webUI host to openSUSE Leap 15.3</i> to <i>Upgrade o3 webUI host to openSUSE Leap 15.3 size:M</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/451146/diff?detail_id=427830">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li></ul> openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4515962021-10-01T13:46:16Zlivdywanliv.dywan@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>livdywan</i></li></ul><p>It occured to me I can do this while poking at running tests, so I'm taking this now</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4516142021-10-01T15:31:10Zlivdywanliv.dywan@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>cdywan wrote:</p>
<blockquote>
<p>It occured to me I can do this while poking at running tests, so I'm taking this now</p>
</blockquote>
<p>Went through the upgrade as per the steps in the wiki, rebooted. Workers seem to have reconnected fine and jobs are running.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4516442021-10-01T23:10:33Zokurzokurz@suse.com
<ul></ul><p>rpmconfigcheck showed one file /etc/postfix/master.cf.rpmnew which I diffed with /etc/postfix/master.cf , took over some updates and then deleted the rpmnew file. I think rest looks really good. Great work!</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4517522021-10-04T08:23:38Zlivdywanliv.dywan@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>okurz wrote:</p>
<blockquote>
<p>rpmconfigcheck showed one file /etc/postfix/master.cf.rpmnew which I diffed with /etc/postfix/master.cf , took over some updates and then deleted the rpmnew file. I think rest looks really good. Great work!</p>
</blockquote>
<p>Arg, so I missed something afterall... Thank you for checking!</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4519142021-10-04T13:52:45Zlivdywanliv.dywan@suse.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Feedback</i></li></ul><p>Apparently I missed something else, too:</p>
<pre><code>Oct 01 14:25:44 ariel openqa-gru[30835]: Can't exec "/bin/sh": Permission denied at /usr/share/openqa/script/../lib/OpenQA/Task/Job/FinalizeResults.pm line 63.
</code></pre>
<p>Keeping in mind that <em>osd</em> uses UTC, this should fit into the time window of when I was wrapping up the upgrade as per my comment above. And there were apparmor changes, which I presumably didn't do correctly.</p>
<p>I also filed <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: Minion jobs for job hooks failed silently on o3 size:M (Resolved)" href="https://progress.opensuse.org/issues/99741">#99741</a> because this didn't trigger any alerts and was discovered by <a class="user active user-mention" href="https://progress.opensuse.org/users/33482">@tinita</a>.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4519172021-10-04T13:57:55Zlivdywanliv.dywan@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/99741">action #99741</a>: Minion jobs for job hooks failed silently on o3 size:M</i> added</li></ul> openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4519352021-10-04T14:25:15Zlivdywanliv.dywan@suse.com
<ul></ul><p>cdywan wrote:</p>
<blockquote>
<p>Apparently I missed something else, too:</p>
<pre><code>Oct 01 14:25:44 ariel openqa-gru[30835]: Can't exec "/bin/sh": Permission denied at /usr/share/openqa/script/../lib/OpenQA/Task/Job/FinalizeResults.pm line 63.
</code></pre>
<p>Keeping in mind that <em>osd</em> uses UTC, this should fit into the time window of when I was wrapping up the upgrade as per my comment above. And there were apparmor changes, which I presumably didn't do correctly.</p>
<p>I also filed <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: Minion jobs for job hooks failed silently on o3 size:M (Resolved)" href="https://progress.opensuse.org/issues/99741">#99741</a> because this didn't trigger any alerts and was discovered by <a class="user active user-mention" href="https://progress.opensuse.org/users/33482">@tinita</a>.</p>
</blockquote>
<p>I reset <code>/etc/apparmor.d/local/usr.share.openqa.script.openqa</code> to a comments-only file, which it should after <a href="https://github.com/os-autoinst/openQA/pull/3847" class="external">os-autoinst/openQA/pull/3847</a> and which I guess is what I mistook for something we needed to keep.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4520102021-10-04T16:28:23Zlivdywanliv.dywan@suse.com
<ul></ul><p>I still can't tell if the files in <code>/etc/apparmor.d/{,local/}usr.share.openqa.script.openqa</code> are correct. And I wasn't able to figure out how to access the most recent copy of <code>/etc/apparmor.d/usr.share.openqa.script.openqa</code>, if one exists, since <a href="https://progress.opensuse.org/projects/openqav3/wiki#Backup" class="external">the Backup section</a> doesn't really explain that and it's pretty much Greek to me if you excuse the pun.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4522112021-10-05T10:04:47Zlivdywanliv.dywan@suse.com
<ul></ul><p>For now I'm monitoring logs to see if the errors persist (via <code>sudo journalctl -f -u openqa-gru</code>). And I added <code>/bin/sh mrix,</code> to <code>/etc/apparmor.d/usr.share.openqa.script.openqa</code>.<br>
Also tried <code>sudo systemctl restart openqa-gru</code>, to no apparent effect. Btw for reference <em>o3</em> is on <em>apparmor-profiles 2.13.6-1.31</em>, as opposed to <em>osd/2.13.4-lp152.2.3.1</em>.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4522772021-10-05T12:57:49Zlivdywanliv.dywan@suse.com
<ul></ul><p>It would seem comparing to <em>osd</em> was pointless since according to <code>sudo aa-status</code> it's currently switched off there 🤦️</p>
<p>Trying to see now if <code>sudo aa-complain /usr/share/openqa/script/openqa{,-cli}</code> yields some more information here.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4523072021-10-05T14:22:46Zlivdywanliv.dywan@suse.com
<ul></ul><blockquote>
<p>/opt/os-autoinst-scripts/openqa-label-known-issues: line 83: hxselect: command not found<br>
grep: write error: Broken pipe</p>
</blockquote>
<p>Not sure if these are related, but while I'm at it I'm installing <code>html-xml-utils</code>.</p>
<p>Btw I also created a proof of concept for <code>dependencies.yaml</code> in the <em>scripts</em> repo, although this will need a bit of polishing before it can be used: <a href="https://github.com/os-autoinst/scripts/pull/116" class="external">https://github.com/os-autoinst/scripts/pull/116</a></p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4523342021-10-05T16:05:54Zlivdywanliv.dywan@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>I'm assuming it's working now since I no longer see errors and I can see investigate jobs that spawned and finished successfully.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4523432021-10-05T17:34:15Ztinitatina.mueller+trick-redmine@suse.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Feedback</i></li></ul><p>cdywan wrote:</p>
<blockquote>
<p>Trying to see now if <code>sudo aa-complain /usr/share/openqa/script/openqa{,-cli}</code> yields some more information here.</p>
</blockquote>
<p>This sets it to complain mode, and any violations are just logged (<a href="https://wiki.ubuntu.com/DebuggingApparmor#Debugging_procedure" class="external">https://wiki.ubuntu.com/DebuggingApparmor#Debugging_procedure</a>)</p>
<p>So if you don't see error messages in the gru journal, that's because it's in complain mode (but I don't know where the "complaints" are actually going to).</p>
<p>So if you didn't do anything else, then this is not a fix.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4523552021-10-05T19:37:52Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>PR for apparmor profile fix: <a href="https://github.com/os-autoinst/openQA/pull/4271" class="external">https://github.com/os-autoinst/openQA/pull/4271</a></p>
<p>In Leap 15.2, <code>/bin/sh</code> points to <code>/bin/bash</code>, while in 15.3,<br>
it points to <code>/usr/bin/sh</code> -> <code>/usr/bin/bash</code></p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4524722021-10-06T07:12:28Zlivdywanliv.dywan@suse.com
<ul></ul><p>tinita wrote:</p>
<blockquote>
<p>PR for apparmor profile fix: <a href="https://github.com/os-autoinst/openQA/pull/4271" class="external">https://github.com/os-autoinst/openQA/pull/4271</a></p>
<p>In Leap 15.2, <code>/bin/sh</code> points to <code>/bin/bash</code>, while in 15.3,<br>
it points to <code>/usr/bin/sh</code> -> <code>/usr/bin/bash</code></p>
</blockquote>
<p>I'm wondering how you confirmed that this worked, since I seem to have seen successfully executed hooks without any errors in the entire journal 🤔️<br>
So I guess to resolve it for good I need to find out where the presumed missing error messages end up, and document it.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4525712021-10-06T08:24:15Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>Sorry, I forgot to add:<br>
I did the mentioned fix locally (add <code>/usr/bin/bash</code>), and then did<br>
<code>aa-enforce /usr/share/openqa/script/openqa</code><br>
to end the complain mode.<br>
Then I saw successful hooks by looking into the <code>minion_jobs</code> table and I didn't see errors in the openqa-gru journal anymore.</p>
<p>Note that if apparmor is in complain mode, one is not supposed to see the error messages, but there will be messages in <code>/var/log/audit/audit.log</code>.</p>
<p>Today I saw new errors though:<br>
<code>/opt/os-autoinst-scripts/openqa-label-known-issues: line 83: /usr/bin/hxselect: Permission denied</code><br>
PR for that: <a href="https://github.com/os-autoinst/openQA/pull/4273" class="external">https://github.com/os-autoinst/openQA/pull/4273</a></p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4527152021-10-06T10:19:05Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>PR <a href="https://github.com/os-autoinst/openQA/pull/4273" class="external">https://github.com/os-autoinst/openQA/pull/4273</a> merged, and I added the line manually on o3 to not wait until the next deployment.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4527182021-10-06T10:21:55Zlivdywanliv.dywan@suse.com
<ul></ul><p>Post mortem:</p>
<ul>
<li>[x] I filed <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: Minion jobs for job hooks failed silently on o3 size:M (Resolved)" href="https://progress.opensuse.org/issues/99741">#99741</a> to address the silent failures</li>
<li>[x] There's also <a class="issue tracker-4 status-12 priority-3 priority-lowest" title="action: Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respect... (Workable)" href="https://progress.opensuse.org/issues/57239">#57239</a> which could have helped spot the problem earlier</li>
<li>[ ] Is there a feature request/bug on AppArmor wrt unclear error message?</li>
<li>[ ] I'll try and propose documentation for how AppArmor is handled with openQA</li>
</ul>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4531892021-10-07T14:03:08Zlivdywanliv.dywan@suse.com
<ul></ul><p>cdywan wrote:</p>
<blockquote>
<ul>
<li>[ ] Is there a feature request/bug on AppArmor wrt unclear error message?</li>
</ul>
</blockquote>
<p><a href="https://gitlab.com/apparmor/apparmor/-/issues/201" class="external">https://gitlab.com/apparmor/apparmor/-/issues/201</a></p>
<blockquote>
<ul>
<li>[ ] I'll try and propose documentation for how AppArmor is handled with openQA</li>
</ul>
</blockquote>
<p><a href="https://github.com/os-autoinst/openQA/pull/4278" class="external">https://github.com/os-autoinst/openQA/pull/4278</a></p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=4579352021-10-22T09:44:08Zlivdywanliv.dywan@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>Including the potential upstream improvements and additions to openQA docs, I think the host is looking good at this point. And of course thanks to <a class="user active user-mention" href="https://progress.opensuse.org/users/33482">@tinita</a> especially.</p>
openQA Infrastructure - action #99195: Upgrade o3 webUI host to openSUSE Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99195?journal_id=5245672022-05-31T18:19:54Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed child" href="/issues/111869">action #111869</a>: Upgrade o3 webUI host to openSUSE Leap 15.4 size:S</i> added</li></ul>