openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-27T08:03:58ZopenSUSE Project Management Tool
Redmine openQA Infrastructure - action #158113 (Feedback): typing issue on ppc64 worker - make CPU load a...https://progress.opensuse.org/issues/1581132024-03-27T08:03:58Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a class="issue tracker-4 status-4 priority-5 priority-high3 child behind-schedule" title="action: typing issue on ppc64 worker size:S (Feedback)" href="https://progress.opensuse.org/issues/158104">#158104</a> shows VNC typing issues. For this in <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: CPU Load and usage alert for openQA workers size:S (Resolved)" href="https://progress.opensuse.org/issues/150983">#150983</a> on purpose we added alerts to alert on too high CPU load. <a href="https://monitor.qa.suse.de/d/WDmania/worker-dashboard-mania?orgId=1&from=now-2d&to=now&viewPanel=54694" class="external">https://monitor.qa.suse.de/d/WDmania/worker-dashboard-mania?orgId=1&from=now-2d&to=now&viewPanel=54694</a> clearly shows a load consistently in the range of 50-70(!) for mania but no alert triggered. We should crosscheck <a href="https://monitor.qa.suse.de/alerting/cpu_load_alert_mania/modify-export?returnTo=%2Fd%2FWDmania%2Fworker-dashboard-mania%3ForgId%3D1%26from%3Dnow-7d%26to%3Dnow%26viewPanel%3D54694%26editPanel%3D54694%26tab%3Dalert" class="external">https://monitor.qa.suse.de/alerting/cpu_load_alert_mania/modify-export?returnTo=%2Fd%2FWDmania%2Fworker-dashboard-mania%3ForgId%3D1%26from%3Dnow-7d%26to%3Dnow%26viewPanel%3D54694%26editPanel%3D54694%26tab%3Dalert</a><br>
and make that alert more strict.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> CPU load alerts trigger for a CPU load15 consistently above 40 as originally planned</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Crosscheck <a href="https://monitor.qa.suse.de/alerting/cpu_load_alert_mania/modify-export?returnTo=%2Fd%2FWDmania%2Fworker-dashboard-mania%3ForgId%3D1%26from%3Dnow-7d%26to%3Dnow%26viewPanel%3D54694%26editPanel%3D54694%26tab%3Dalert" class="external">https://monitor.qa.suse.de/alerting/cpu_load_alert_mania/modify-export?returnTo=%2Fd%2FWDmania%2Fworker-dashboard-mania%3ForgId%3D1%26from%3Dnow-7d%26to%3Dnow%26viewPanel%3D54694%26editPanel%3D54694%26tab%3Dalert</a> or the implementation in code <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/blame/master/monitoring/grafana/alerting-dashboard-WD.yaml.template?ref_type=heads#L941" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/blame/master/monitoring/grafana/alerting-dashboard-WD.yaml.template?ref_type=heads#L941</a></li>
</ul>
openQA Infrastructure - action #158104 (Feedback): typing issue on ppc64 worker size:Shttps://progress.opensuse.org/issues/1581042024-03-27T06:57:56Zzcjiazcjia@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-15-SP6-Online-ppc64le-ha_beta_supportserver@ppc64le-2g fails in<br>
<a href="https://openqa.suse.de/tests/13885455/modules/setup/steps/84" class="external">setup</a></p>
<p><a href="https://openqa.suse.de/tests/13885455#step/setup/84" class="external">https://openqa.suse.de/tests/13885455#step/setup/84</a> (see attachment p1.png)</p>
<p><a href="https://openqa.suse.de/tests/13885471#step/setup/30" class="external">https://openqa.suse.de/tests/13885471#step/setup/30</a> (see attachment p2.png) It missed "$" before "?".</p>
<p><a href="https://openqa.suse.de/tests/13885404#step/setup/12" class="external">https://openqa.suse.de/tests/13885404#step/setup/12</a> (see attachment p3.png)</p>
<p><a href="https://openqa.suse.de/tests/13885407#step/setup/9" class="external">https://openqa.suse.de/tests/13885407#step/setup/9</a> (see attachment p4.png)</p>
<p>I think this may related with the high work load of underlying ppc64 worker.</p>
<p>All on "mania"</p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>The base test suite is used for job templates defined in YAML documents. It has no settings of its own.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/13885455" class="external">73.1</a> (current job)</p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/13829359" class="external">67.1</a> (or more recent)</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Identify the affected machines and workers, apply mitigations to prevent recurring typing issues, e.g. reducing CPU load</li>
<li>Restart related failed jobs</li>
<li>Identify follow-up tasks</li>
<li>Reduce the number of worker instances as a first mitigation measure. <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/759" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/759</a> (merged)</li>
<li>Make the alert for CPU load more strict - <a class="issue tracker-4 status-4 priority-5 priority-high3 child behind-schedule" title="action: typing issue on ppc64 worker - make CPU load alert more strict (Feedback)" href="https://progress.opensuse.org/issues/158113">#158113</a></li>
<li>Evaluate the impact on video encoding in particular on ppc64le, maybe ffmpeg on Power8 kvm is inefficient - <a class="issue tracker-4 status-1 priority-4 priority-default child" title="action: typing issue on ppc64 worker - crosscheck performance impact of ffmpeg on ppc64le (Power8 kvm) (New)" href="https://progress.opensuse.org/issues/158116">#158116</a></li>
<li>Check existing ffmpeg processes on mania which take a lot of CPU time - <a class="issue tracker-4 status-1 priority-4 priority-default child" title="action: typing issue on ppc64 worker - crosscheck performance impact of ffmpeg on ppc64le (Power8 kvm) (New)" href="https://progress.opensuse.org/issues/158116">#158116</a></li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>ffmpeg impact investigation -> <a class="issue tracker-4 status-4 priority-5 priority-high3 child behind-schedule" title="action: typing issue on ppc64 worker - make CPU load alert more strict (Feedback)" href="https://progress.opensuse.org/issues/158113">#158113</a></li>
<li>code improvements -> <a class="issue tracker-4 status-1 priority-4 priority-default child" title="action: typing issue on ppc64 worker - only pick up (or start) new jobs if CPU load is below configured t... (New)" href="https://progress.opensuse.org/issues/158125">#158125</a></li>
<li>improving the alert -> <a class="issue tracker-4 status-4 priority-5 priority-high3 child behind-schedule" title="action: typing issue on ppc64 worker - make CPU load alert more strict (Feedback)" href="https://progress.opensuse.org/issues/158113">#158113</a></li>
</ul>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=ppc64le&distri=sle&flavor=Online&machine=ppc64le-2g&test=ha_beta_supportserver&version=15-SP6" class="external">latest</a></p>
QA - action #157858 (Feedback): Repeated reminder comments about SLO's for openqatests size:Shttps://progress.opensuse.org/issues/1578582024-03-25T08:37:52Zlivdywanliv.dywan@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: No ticket reminder comments about SLO's for openqatests size:M (Resolved)" href="https://progress.opensuse.org/issues/157522">#157522</a> addressed a bug that prevented reminder comments from being sent. Unfortunately comments are added even if a comment was already present. This is especially visible in <em>immediate</em> tickets, for example #153115, which get daily reminders - as per <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Automated alerts and reminders about SLO's for openqatests (only one reminder) size:M (Resolved)" href="https://progress.opensuse.org/issues/116545">#116545</a> only one comment is supposed to be added. Maybe this is a regression or the check is not comprehensive enough.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Reminders are only added once</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>We already have the code that should handle that: Review the implementation from <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Automated alerts and reminders about SLO's for openqatests (only one reminder) size:M (Resolved)" href="https://progress.opensuse.org/issues/116545">#116545</a> for gaps in the current logic in <a href="https://github.com/openSUSE/backlogger/blob/main/backlogger.py" class="external">https://github.com/openSUSE/backlogger/blob/main/backlogger.py</a></li>
<li>Investigate if something changed with current comments, maybe the Redmine upgrade made a difference here (complete guess)?</li>
<li>Maybe the regex needs to be adapted and/or better covered with unit testing</li>
</ul>
openQA Project - action #157540 (Feedback): [sporadic] ci openQA: t/33-developer_mode.t fails size:Mhttps://progress.opensuse.org/issues/1575402024-03-19T14:15:50Ztinitatina.mueller+trick-redmine@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://app.circleci.com/pipelines/github/os-autoinst/openQA/13196/workflows/ddb935c7-31dd-4beb-877c-25ef1e703b4d/jobs/123231" class="external">https://app.circleci.com/pipelines/github/os-autoinst/openQA/13196/workflows/ddb935c7-31dd-4beb-877c-25ef1e703b4d/jobs/123231</a></p>
<pre><code>[14:03:42] t/33-developer_mode.t .. 17/? # Unexpected Javascript console errors, waiting for connection opened: [
# {
# level => "SEVERE",
# message => "http://localhost:9526/asset/3906633cf0/ws_console.js 8 WebSocket connection to 'ws://localhost:9528/liveviewhandler/tests/1/developer/ws-proxy' failed: Error during WebSocket handshake: Unexpected response code: 302",
# source => "network",
# timestamp => 1710857067816,
# },
# ]
# Failed test 'No unexpected js warnings'
# at /home/squamata/project/t/lib/OpenQA/Test/FullstackUtils.pm line 123.
# Looks like you failed 1 test of 9.
[14:03:42] t/33-developer_mode.t .. 20/?
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>While investigating the code in parallel try to reproduce locally with coverage enabled and multiple runs to get a statistically significant result, e.g. <code>make test KEEP_DB=1 RETRY=500 TESTS=t/33-developer.t</code> and go for lunch or continue coding :)</li>
<li>If it's not reproducible consider the same with coverage enabled and/or in circleCI, e.g. a temporary branch in your github repo fork</li>
<li>Identify where in <a href="https://github.com/os-autoinst/openQA/blob/master/t/33-developer_mode.t" class="external">https://github.com/os-autoinst/openQA/blob/master/t/33-developer_mode.t</a> the redirection "302" could happen</li>
<li>Even though the test is not technically a UI test in the t/ui/ folder it might still be necessary to apply UI test related synchronisation means to fix the sporadic failure as a selenium instance is used</li>
<li>Might be a similar issue: <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [sporadic] t/full-stack.t Failed test 'Expected result for job 1 not found' size:M (Resolved)" href="https://progress.opensuse.org/issues/102578">#102578</a></li>
</ul>
openQA Project - coordination #157537 (Blocked): [epic] Secure setup of openQA test machines with...https://progress.opensuse.org/issues/1575372024-03-19T14:15:29Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>In <a href="https://sd.suse.com/servicedesk/customer/portal/1/SD-150437" class="external">https://sd.suse.com/servicedesk/customer/portal/1/SD-150437</a> we are asked to handle "compromised root passwords in QA segments" including s390zl11…16 . We should secure our network and password handling better.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> No openQA machine test machines directly accessible by SUSE users use ssh root with publically known passwords</li>
</ul>
<a name="Ideas"></a>
<h2 >Ideas<a href="#Ideas" class="wiki-anchor">¶</a></h2>
<ol>
<li>Be able to set a different password valid for tests, in particular s390kvm…, e.g. be able to set password by test variable and follow through in the complete test platform -> <a class="issue tracker-4 status-12 priority-5 priority-high3 child" title="action: [spike][timeboxed:10h] Use a different ssh root password for s390x kvm installation openQA jobs (... (Workable)" href="https://progress.opensuse.org/issues/157555">#157555</a></li>
<li>Key based authentication -> <a class="issue tracker-4 status-15 priority-4 priority-default child" title="action: [spike][timeboxed:10h] Use ssh key authentication in particular for s390x kvm installation openQA... (Blocked)" href="https://progress.opensuse.org/issues/157744">#157744</a></li>
<li>Rotating, automatic passwords saved as test variables connected to images, e.g. to be able to use a pre-installed image</li>
<li>Better secure the networks to have s390kvm… (and others) less accessible -> We have stated the requirement in <a href="https://confluence.suse.com/pages/viewpage.action?pageId=1006108843" class="external">https://confluence.suse.com/pages/viewpage.action?pageId=1006108843</a> that ssh 22/tcp needs to be reachable. We could try to replicate the setup we know from o3 to give OSD a second network interface which allows ssh 22/tcp and block ssh 22/tcp on .oqa.prg2.suse.org as usually we don't need ssh to workers, just from within the oqa network as well as for administrative purposes for which we could go over OSD which we also already normally do for salt. -> <a class="issue tracker-4 status-1 priority-4 priority-default child" title="action: Better secure the networks to have s390kvm… (and others) less accessible (New)" href="https://progress.opensuse.org/issues/157750">#157750</a></li>
</ol>
openQA Project - action #157369 (Feedback): Handle all node dependabot updates, not just security...https://progress.opensuse.org/issues/1573692024-03-15T21:04:40Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>With #155410 resolved we have dependabot updates in <a href="https://github.com/os-autoinst/openQA/" class="external">https://github.com/os-autoinst/openQA/</a>, actually already for all node updates, not just security updates. But we need to help dependabot on getting the updates done, e.g. update our code and tests so that they cope with a newer version. For trivial cases we already have dependabot creating the pull request and mergify eventually merging it after a wait time of multiple days. For the cases where CI tests fail we need people to push code changes. Maybe just mention it on <a href="https://progress.opensuse.org/projects/qa/wiki/tools" class="external">https://progress.opensuse.org/projects/qa/wiki/tools</a> that we should support such pull requests, set aside work time to support those updates and in cases where it's becoming too much effort just create an according ticket for each pull request that needs more work.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> The team is confident how to handle dependabot updates as part of their daily work</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Add on an appropriate place on progress.opensuse.org/projects/qa/wiki/tools how to handle such updates</li>
<li>Tell everyone from the team, ask them for feedback, adjust</li>
</ul>
openQA Project - action #156553 (Blocked): [timeboxed:10h][spike solution] openQA webUI search vi...https://progress.opensuse.org/issues/1565532024-03-04T11:07:42Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>From #121246-15: "We'd need to look for all the tests that are failing for a given incident, using the same TEST_ISSUES for both, Aggregates and Incidents". So what is needed is a single command line or openQA webUI search view to show all tests blocking an incident by squad. After <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655">#117655</a> and <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: [spike][timeboxed:20h] Filter openQA todo-jobs on /tests belonging to one "review squad" size:S (Resolved)" href="https://progress.opensuse.org/issues/119746">#119746</a> and <a class="issue tracker-4 status-12 priority-4 priority-default child" title="action: A single API route to show all not-ok tests blocking a SLE maintenance incident size:M (Workable)" href="https://progress.opensuse.org/issues/156547">#156547</a> we should combine both.</p>
<a name="Goals"></a>
<h2 >Goals<a href="#Goals" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>G1:</strong> Proof-of-concept for an openQA webUI search view to show all tests blocking an incident by squad, e.g. based on special job setting or group glob</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>We have support for group globbing (<a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Filter openQA todo-jobs on /tests belonging to one "review squad" size:M (Resolved)" href="https://progress.opensuse.org/issues/134933#note-32">#134933#note-32</a>)
<ul>
<li><a href="https://openqa.opensuse.org/tests?group_glob=*Leap*&todo=1" class="external">https://openqa.opensuse.org/tests?group_glob=*Leap*&todo=1</a></li>
</ul></li>
<li>"squads" could be mapped into openQA for example with special job settings, e.g. QE Core ensures to trigger all their tests with _SQUAD='QE Core' and then be able to filter by that</li>
<li>This doesn't need to be specific to squads/blocking tests (openQA itself should not know about these SUSE specific concepts)</li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>We don't care if searching for job settings is limited by an artificial search depth or super slow -> <a class="issue tracker-4 status-12 priority-4 priority-default child" title="action: A single API route to show all not-ok tests blocking a SLE maintenance incident size:M (Workable)" href="https://progress.opensuse.org/issues/156547">#156547</a></li>
</ul>
QA - action #153733 (Feedback): Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - soapberryhttps://progress.opensuse.org/issues/1537332024-01-16T20:12:28Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> soapberry is usable from PRG2</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Follow <a href="https://jira.suse.com/browse/ENGINFRA-3748" class="external">https://jira.suse.com/browse/ENGINFRA-3748</a></li>
<li>Ensure machine can be reached</li>
<li>Ensure machine is used as in before migration</li>
</ul>
QA - action #153724 (Feedback): Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - blackcur...https://progress.opensuse.org/issues/1537242024-01-16T20:07:13Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> blackcurrant is usable from PRG2</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Follow <a href="https://jira.suse.com/browse/ENGINFRA-3745" class="external">https://jira.suse.com/browse/ENGINFRA-3745</a></li>
<li>Ensure machine can be reached</li>
<li>Ensure machine is used as in before migration</li>
</ul>
QA - action #153718 (Feedback): Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - haldirhttps://progress.opensuse.org/issues/1537182024-01-16T20:02:28Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> haldir is usable from PRG2</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Follow <a href="https://jira.suse.com/browse/ENGINFRA-3744" class="external">https://jira.suse.com/browse/ENGINFRA-3744</a></li>
<li>Ensure machine can be reached</li>
<li>Ensure machine is used as in before migration</li>
</ul>
openQA Project - coordination #152847 (Blocked): [epic] version control awareness within openQA f...https://progress.opensuse.org/issues/1528472023-12-21T12:48:46Zokurzokurz@suse.comQA - coordination #129280 (Blocked): [epic] Move from SUSE NUE1 (Maxtorhof) to new NBG Datacentershttps://progress.opensuse.org/issues/1292802023-05-15T07:12:10Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>SUSE NUE1 is being evacuated so we need to ensure our services are provided from other places and that NUE1 has been evacuated by us.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> NUE1 (Maxtorhof) is not relied upon by SUSE QE Tools anymore and has been evacuated by us</li>
</ul>
<a name="Ideas"></a>
<h2 >Ideas<a href="#Ideas" class="wiki-anchor">¶</a></h2>
<ul>
<li>"To-be-decommissioned" machines obviously should not be moved to a new datacenter</li>
<li>Consider decommissioning some more machines in the process, e.g. "qanet" which should be replaced by Eng-Infra maintained DHCP+DNS same as we have in PRG1, PRG2, NUE2 (e.g. FC Basement) and also qanet does not have proper remote management capabilities</li>
<li>Some machines might be better moved to FC Basement rather than new NBG Datacenter</li>
</ul>
QA - coordination #123800 (Blocked): [epic] Provide SUSE QE Tools services running in PRG2 aka. P...https://progress.opensuse.org/issues/1238002023-01-30T14:46:55Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>SUSE is deprecating NUE1 (Maxtorhof) and setting up a Prague Co-Location datacenter "Prg CoLo" or "DC7" as primary location in particular for serving public services. This includes what we serve so far from VM clusters managed by EngInfra and in particular the openqa.opensuse.org infrastructure, likely also openqa.suse.de. We must participate in planning and setup and accordingly a migration until we can provide our services from Prg CoLo.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> SUSE QE Tools services are provided out of Prg CoLo</li>
</ul>
QA - coordination #121720 (Blocked): [saga][epic] Migration to QE setup in PRG2+NUE3 while ensuri...https://progress.opensuse.org/issues/1217202022-12-08T19:30:27Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>SUSE is deprecating NUE1 (Maxtorhof) and setting up a Prague Co-Location datacenter "Prg CoLo" or "DC7" as primary location in particular for serving public services. This includes what we serve so far from VM clusters managed by EngInfra and in particular the openqa.opensuse.org infrastructure, likely also openqa.suse.de. Or defined differently: Everything that is currently served from NUE1-SRV1. We must participate in planning and setup and accordingly a migration until we can provide our services from Prg CoLo and do not rely on NUE1-SRV1 anymore except for the purpose of an optional fail-over datacenter in Nbg.<br>
SUSE is deprecating NUE1 (Maxtorhof) and setting up replacement data centers. Additionally a new datacenter is planned as fail-over location</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> SUSE QE Tools services are provided out of Prg CoLo <a class="issue tracker-6 status-15 priority-4 priority-default child parent behind-schedule" title="coordination: [epic] Provide SUSE QE Tools services running in PRG2 aka. Prg CoLo (Blocked)" href="https://progress.opensuse.org/issues/123800">#123800</a></li>
<li><strong>AC2:</strong> NUE1 (Maxtorhof) is not relied upon by SUSE QE Tools anymore and has been evacuated by us <a class="issue tracker-6 status-15 priority-4 priority-default child parent behind-schedule" title="coordination: [epic] Move from SUSE NUE1 (Maxtorhof) to new NBG Datacenters (Blocked)" href="https://progress.opensuse.org/issues/129280">#129280</a></li>
<li><strong>AC3:</strong> Relevant SUSE QE Tools services are provided out of NUE3 <a class="issue tracker-6 status-3 priority-4 priority-default closed child parent" title="coordination: [epic] Migration out of SUSE NUE1 - QE setup in NUE3 (Resolved)" href="https://progress.opensuse.org/issues/130955">#130955</a></li>
</ul>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Coordination chat room <a href="https://suse.slack.com/archives/C04MDKHQE20" class="external">#dct-migration</a></p>
openQA Project - coordination #58184 (Blocked): [saga][epic][use case] full version control aware...https://progress.opensuse.org/issues/581842019-10-15T10:19:57Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>This is linked to <a href="https://progress.opensuse.org/projects/openqav3/wiki#Use-case-4" class="external">Use case 4</a> and motivated by a discussion by the QA tools team in the weekly meeting 2019-10-15. What we should have are for example user forks and branches, fully versioned test schedules and configuration settings</p>
<a name="User-story"></a>
<h2 >User story<a href="#User-story" class="wiki-anchor">¶</a></h2>
<p>As a test case contributor during test case development I want to run tests on production instances with all necessary changes recorded in version control before merging to master so that my change will have minimal unexpected impact (test regressions) on existing tests</p>
<a name="Further-user-stories-from-httpsconfluencesusecompagesviewpageactionpageId365527173"></a>
<h2 >Further user stories (from <a href="https://confluence.suse.com/pages/viewpage.action?pageId=365527173" class="external">https://confluence.suse.com/pages/viewpage.action?pageId=365527173</a>)<a href="#Further-user-stories-from-httpsconfluencesusecompagesviewpageactionpageId365527173" class="wiki-anchor">¶</a></h2>
<ol>
<li>I want to start a job based on a modified test in production (In production tests can behave differently, for example because of the heavier load) -> see openqa-clone-job + CASEDIR</li>
<li>I want to edit needles and test if they work before proposing changes</li>
<li>I want to compare the results of a certain job group between two of my branches</li>
<li>I want to schedule a test 100 times without it showing up in the group overview -> see <a href="https://progress.opensuse.org/projects/openqatests/wiki#Statistical-investigation" class="external">statistical-investigation</a></li>
<li>I want to trigger multiple cloned jobs for each pull-request (Sometimes you want to trigger VR for different jobs against the same PR. it would be nice to do that in one command line)</li>
<li>I want to trigger the relevant tests automatically by creating a PR</li>
</ol>
<a name="Implications-and-suggestions"></a>
<h2 >Implications and suggestions<a href="#Implications-and-suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><p>The usual test contributor workflows should be supported and made easier by making openQA fully aware of tests triggered for development purposes without negatively impacting existing validation tests</p>
<ul>
<li>Potential impact on asset management</li>
<li>No pollution of validation test reports by development tests</li>
</ul></li>
<li><p>If there are new/modified needles involved, the existing workflow cannot handle that. The current practice is:</p>
<ul>
<li>Test your changes (and possibly needle changes) locally and create PR(s)</li>
<li>Edit needles online and save them (then they will be committed to master). Requires admin rights</li>
</ul></li>
<li><p>DONE: Cloning cancelled or incomplete jobs currently does not work as openqa-clone-custom-git-refspec requires the vars.json file from a completed job with this file uploaded -> <a href="https://github.com/os-autoinst/openQA/pull/3170" class="external">https://github.com/os-autoinst/openQA/pull/3170</a></p></li>
<li><p>Replace "fetchneedles" by inherent git support</p></li>
<li><p>Provide support for github pull request validation</p></li>
<li><p>DONE: Extend openqa-clone-custom-git-refspec to accept list of source tests to clone -> <a href="https://github.com/os-autoinst/openQA/pull/2577" class="external">https://github.com/os-autoinst/openQA/pull/2577</a></p></li>
<li><p>DONE: openqa-clone-custom-git-refspec: Output in markdown format for easy copy/pasting into git commit messages and github PR comments -> <a href="https://github.com/os-autoinst/openQA/pull/2577" class="external">https://github.com/os-autoinst/openQA/pull/2577</a></p></li>
<li><p>openqa-clone-custom-git-refspec: Provide link to /tests/overview page for the custom build when multiple tests have been cloned</p></li>
<li><p>Make the trigger source of test jobs apparent, e.g. the source git repositories</p></li>
<li><p><a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="coordination: [EPIC] Interactive mode is an usability disaster (Resolved)" href="https://progress.opensuse.org/issues/14818#note-18">#14818#note-18</a> : "Tim got a ticket from Ray that the docker test failed and wants openQA to reproduce the issue and pause at the beginning of the docker test. Afterwards he wants openQA to make a disk snapshot and step through the test execution to find out where the problem is. After he found out, he reloads the snapshot to tweak the execution. During this process, openQA records his steps and allows to add needles."</p></li>
</ul>