openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-25T15:12:22ZopenSUSE Project Management Tool
Redmine openQA Project - action #157912 (Resolved): Scheduling a product can fail on asset creation becau...https://progress.opensuse.org/issues/1579122024-03-25T15:12:22Zmkittlermarius.kittler@suse.com
<a name="Observation"></a>
<h3 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h3>
<pre><code>{
"failed_job_info": [
{
"error_message": "DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: duplicate key value violates unique constraint \"assets_type_name\"\nDETAIL: Key (type, name)=(hdd, autoyast_SLES-12-SP5-x86_64-create_hdd_yast_maintenance_desktop-Build20240324-1.qcow2) already exists. [for Statement \"INSERT INTO assets ( name, t_created, t_updated, type) VALUES ( ?, ?, ?, ? ) RETURNING id\" with ParamValues: 1='autoyast_SLES-12-SP5-x86_64-create_hdd_yast_maintenance_desktop-Build20240324-1.qcow2', 2='2024-03-24 20:07:45', 3='2024-03-24 20:07:45', 4='hdd'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1535\n",
"job_name": "migr_sles12sp5_"
},
…
{
"error_messages": [
"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: current transaction is aborted, commands ignored until end of transaction block [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='13854606', 2='1', 3='13854599'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 721\n"
],
"job_id": 13854599
}
],
"notes": [
"Transaction failed: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: current transaction is aborted, commands ignored until end of transaction block [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='13854606', 2='1', 3='13854599'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 721\n"
],
"successful_job_ids": []
</code></pre>
<p>See <a href="https://openqa.suse.de/admin/productlog?id=2081528" class="external">https://openqa.suse.de/admin/productlog?id=2081528</a> and <a href="https://suse.slack.com/archives/C02CANHLANP/p1711363172886899" class="external">https://suse.slack.com/archives/C02CANHLANP/p1711363172886899</a></p>
<a name="Acceptance-criteria"></a>
<h3 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h3>
<ul>
<li><strong>AC1</strong>: When scheduling a product openQA doesn't run into errors like "duplicate key value violates unique constraint", even when the same asset is registered concurrently.</li>
</ul>
<a name="Suggestions"></a>
<h3 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h3>
<ul>
<li>Replace <code>my $asset = $assets->find_or_create($asset_info);</code> (currently at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1535) with raw SQL using <code>ON CONFLICT DO NOTHING</code>.</li>
<li>Run e.g. <code>t/api/04-jobs.t</code> to test the changes.</li>
</ul>
openQA Project - action #157543 (Resolved): [sporadic] ci openQA: t/ui/23-audit-log.t fails size:Mhttps://progress.opensuse.org/issues/1575432024-03-19T14:16:41Ztinitatina.mueller+trick-redmine@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://app.circleci.com/pipelines/github/os-autoinst/openQA/13196/workflows/ddb935c7-31dd-4beb-877c-25ef1e703b4d/jobs/123228" class="external">https://app.circleci.com/pipelines/github/os-autoinst/openQA/13196/workflows/ddb935c7-31dd-4beb-877c-25ef1e703b4d/jobs/123228</a></p>
<pre><code>[14:08:28] t/ui/23-audit-log.t ........................ 12/?
# Failed test 'most rows filtered out when searching for table create events'
# at t/ui/23-audit-log.t line 40.
# got: '8'
# expected: '3'
# Looks like you failed 1 test of 22.
[14:08:28] t/ui/23-audit-log.t ........................ 13/?
# Failed test 'clickable events'
# at t/ui/23-audit-log.t line 152.
[14:08:28] t/ui/23-audit-log.t ........................ 14/? # Looks like you failed 1 test of 14.
[14:08:28] t/ui/23-audit-log.t ........................ Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/14 subtests
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Statistically significant stable test execution of t/ui/23-audit-log.t</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>DONE: <del>Find out current error rate locally</del> not reproducible</li>
<li>Find out current error rate locally with coverage enabled as this is possibly more likely to reproduce problems we see in circleCI</li>
<li>Consider recent javascript stack related updates which might impact that</li>
<li>Identify the specific point of sporadic failure source by debugging the unit test and executed code itself</li>
<li>Apply changes to the test code to make it more robust. Possibly similar as in other UI tests in the past with some means of synchronization</li>
<li>Verify that the test is stable again</li>
</ul>
openQA Infrastructure - action #157438 (Resolved): Failed systemd services alert (jenkins-plugins...https://progress.opensuse.org/issues/1574382024-03-18T09:32:00Ztinitatina.mueller+trick-redmine@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<pre><code>Date: Sun, 17 Mar 2024 03:56:33 +0100
1 firing alert instance
[IMAGE]
1 firing instances
Firing [stats.openqa-monitor.qa.suse.de]
Failed systemd services alert (except openqa.suse.de)
View alert [stats.openqa-monitor.qa.suse.de]
Values
B0=1
Labels
alertname
Failed systemd services alert (except openqa.suse.de)
grafana_folder
Salt
rule_uid
Uk02cifVkz
Annotations
message
Check failed systemd services on hosts with `systemctl --failed`. Hint: Go to parent dashboard https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services to see a list of affected hosts.
Silence [stats.openqa-monitor.qa.suse.de]
View dashboard [stats.openqa-monitor.qa.suse.de]
View panel [stats.openqa-monitor.qa.suse.de]
</code></pre>
<p>2024-03-18 10:27:30 <br>
jenkins <br>
jenkins-plugins-update, snapper-cleanup</p>
openQA Infrastructure - action #156934 (Resolved): RPi realhw tests fail with # Test died: Error ...https://progress.opensuse.org/issues/1569342024-03-08T14:53:05Zzluo
<p>e.g. <a href="http://openqa.suse.de/tests/13725224#" class="external">http://openqa.suse.de/tests/13725224#</a></p>
openQA Infrastructure - action #156913 (Resolved): Remove its=off setting in global QEMUMACHINE f...https://progress.opensuse.org/issues/1569132024-03-08T12:03:03Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<ul>
<li>In <a href="https://openqa.suse.de/tests/overview?distri=sle-micro&version=5.1&version=5.2&version=5.3&version=5.4&version=5.5&groupid=535&test=slem_installation_autoyast&arch=aarch64" class="external">test suites</a> there is a <a href="https://bugzilla.suse.com/show_bug.cgi?id=1178033" class="external">Soft Failure</a> due the error <code>install kernel: ITS@0x8080000: Unable to locate ITS domain handle</code></li>
<li>According with this <a href="https://bugzilla.suse.com/show_bug.cgi?id=1178033#c37" class="external">comment</a>, QEMU have to start with <code>-machine virt,usb=off,gic-version=3,its=on</code></li>
<li>Trying to override this value in <code>QEMUMACHINE</code> either in <u>cloned job</u> or <u>isos post</u>, it does not work: the value is still override by <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls" class="external">workerconf.sls</a></li>
</ul>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: remove <code>its=off</code> inside <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls" class="external">workerconf.sls</a>
for <code>worker-arm1</code> and <code>worker-arm2</code> in global <code>QEMUMACHINE</code>:</li>
</ul>
<pre><code class="yaml syntaxhl" data-language="yaml"> <span class="na">worker-arm1</span><span class="pi">:</span>
<span class="c1">## FQDN: worker-arm1.oqa.prg2.suse.org</span>
<span class="c1">## serial: `ssh -t jumpy@qe-jumpy.prg2.suse.org "ipmitool -I lanplus -H openqaworker-arm1.qe-ipmi-ur -U … -P '…' sol activate"`</span>
<span class="na">numofworkers</span><span class="pi">:</span> <span class="m">40</span>
<span class="na">bridge_iface</span><span class="pi">:</span> <span class="s">eth0</span>
<span class="na">webuis</span><span class="pi">:</span>
<span class="na">openqa.suse.de</span><span class="pi">:</span>
<span class="na">key</span><span class="pi">:</span> <span class="s">…</span>
<span class="na">secret</span><span class="pi">:</span> <span class="s">…</span>
<span class="na">global</span><span class="pi">:</span>
<span class="na">WORKER_CLASS</span><span class="pi">:</span> <span class="s">qemu_aarch64,tap,region-prg,location-prg2</span>
<span class="na">QEMUMACHINE</span><span class="pi">:</span> <span class="s">virt,usb=off,gic-version=3,its=off <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<</span>
<span class="na">worker-arm2</span><span class="pi">:</span>
<span class="c1">## FQDN: worker-arm2.oqa.prg2.suse.org</span>
<span class="c1">## serial: `ssh -t jumpy@qe-jumpy.prg2.suse.org "ipmitool -I lanplus -H openqaworker-arm2.qe-ipmi-ur -U … -P '…' sol activate"`</span>
<span class="na">numofworkers</span><span class="pi">:</span> <span class="m">40</span>
<span class="na">bridge_iface</span><span class="pi">:</span> <span class="s">eth0</span>
<span class="na">webuis</span><span class="pi">:</span>
<span class="na">openqa.suse.de</span><span class="pi">:</span>
<span class="na">key</span><span class="pi">:</span> <span class="s">…</span>
<span class="na">secret</span><span class="pi">:</span> <span class="s">…</span>
<span class="na">global</span><span class="pi">:</span>
<span class="na">WORKER_CLASS</span><span class="pi">:</span> <span class="s">qemu_aarch64,tap,region-prg,location-prg2</span>
<span class="na">QEMUMACHINE</span><span class="pi">:</span> <span class="s">virt,usb=off,gic-version=3,its=off <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<</span>
</code></pre>
<a name="How-to-verify"></a>
<h2 >How to verify<a href="#How-to-verify" class="wiki-anchor">¶</a></h2>
<ul>
<li>Restarting the <a href="https://openqa.suse.de/tests/overview?groupid=529" class="external">jobs</a>, we aspect to see inside <code>file/vars.json</code> the value <code>its=on</code> for <code>"QEMUMACHINE"</code></li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Let's make it easy and assume we don't need the QEMUMACHINE settings anymore at all so let's just remove them and verify with openQA jobs that jobs are still fine without</li>
<li>Trigger another specific verification with <code>QEMUMACHINE='its=on'</code></li>
</ul>
<a name="Additional-info"></a>
<h2 >Additional info<a href="#Additional-info" class="wiki-anchor">¶</a></h2>
<ul>
<li>See related ticket <a href="https://progress.opensuse.org/issues/153625" class="external">poo#153625</a></li>
<li>See related bug <a href="https://bugzilla.suse.com/show_bug.cgi?id=1178033" class="external">bsc#1178033</a></li>
<li>See <a href="https://openqa.suse.de/tests/overview?build=issues-153625" class="external">VRs</a></li>
</ul>
openQA Project - action #156907 (Resolved): [tools][qe-core][leap15.6 Beta]test fails in openqa_b...https://progress.opensuse.org/issues/1569072024-03-08T09:34:50Zrfan1richard.fan@suse.com
<a name="Description"></a>
<h2 >Description<a href="#Description" class="wiki-anchor">¶</a></h2>
<p><code>/usr/share/openqa/script/openqa-bootstrap</code> command seems add non existing repo with Beta build.</p>
<p>I didn't check the detail script yet, but seems it has something to do with output of <code>uname -r</code> or <code>uname -m</code> command.</p>
<p>I can see below messages in login console:</p>
<pre><code>Welcome to openSUSE Leap 15.6 Beta - Kernel 6.4.0-150600.9-default (hvc0).
</code></pre>
<p><a href="https://openqa.opensuse.org/tests/3996597#step/openqa_bootstrap/14" class="external">https://openqa.opensuse.org/tests/3996597#step/openqa_bootstrap/14</a> shows that openqa-bootstrap tries to add a repository with content "15.6 Beta" in the URL</p>
<p>So, my questions are:</p>
<ol>
<li>Does it make sense we test this module in Beta phase?</li>
<li>Can we un-schedule this test in Beta phase? [I can change the test code to skip this test then]</li>
</ol>
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario opensuse-15.6-DVD-Updates-x86_64-openqa_bootstrap@64bit fails in<br>
<a href="https://openqa.opensuse.org/tests/3996597/modules/openqa_bootstrap/steps/14" class="external">openqa_bootstrap</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Maintainer: dheidler. Install openQA using openqa-bootstrap script.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.opensuse.org/tests/3996557" class="external">20240308-1</a></p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Possibly <a href="https://build.opensuse.org/project/show/devel:openQA:Leap:15.6" class="external">https://build.opensuse.org/project/show/devel:openQA:Leap:15.6</a> is the correct build repository that should be used. As necessary to fix tests add according dependencies there to fix</li>
</ul>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD-Updates&machine=64bit&test=openqa_bootstrap&version=15.6" class="external">latest</a></p>
openQA Project - action #156754 (Resolved): "DBIx::Class::Row::update(): Can't update OpenQA::Sch...https://progress.opensuse.org/issues/1567542024-03-06T11:43:54Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>As seen in <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: [alert] "HTTP Response" alert fired shortly on 2024-02-12 and 2024-03-04 size:M (Resolved)" href="https://progress.opensuse.org/issues/155326">#155326</a></p>
<p>OSD journal logs show some DBIx error:</p>
<pre><code>Feb 12 00:38:12 openqa openqa[11635]: [error] [ztQJ1_pAsMiS] DBIx::Class::Row::update(): Can't update OpenQA::Schema::Result::JobLocks=HASH(0x55b77ea45e28): row not found at /usr/share/openqa/script/../lib/OpenQA/Resource/Locks.pm line 139
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Specifically look into the only error in the log excerpt "[ztQJ1_pAsMiS] DBIx::Class::Row::update(): Can't update OpenQA::Schema::Result::JobLocks=HASH(0x55b77ea45e28): row not found at /usr/share/openqa/script/../lib/OpenQA/Resource/Locks.pm line 139"</li>
</ul>
openQA Infrastructure - action #156514 (Resolved): Cron <root@openqa-service> (date; fetch_openqa...https://progress.opensuse.org/issues/1565142024-03-04T07:37:00Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Cron <a href="mailto:root@openqa-service">root@openqa-service</a> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log at Mon, 4 Mar 2024 00:20:40 +0000 (UTC):</p>
<pre><code>Exception occured while fetching bsc#1158056
Traceback (most recent call last):
File "/usr/bin/fetch_openqa_bugs", line 62, in <module>
raise e
File "/usr/bin/fetch_openqa_bugs", line 48, in <module>
issue = issue_fetcher.get_issue(bugid)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 88, in get_issue
return self.prefix_table[prefix](self.conf, bugid)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 24, in __init__
self.fetch(conf)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/bugzilla_issue.py", line 28, in fetch
data = req.json()
File "/usr/lib/python3.6/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate what is causing the JSON error - likely an error string was returned instead of JSON</li>
<li>Handle the exception in bugzilla_issue.py and print a meaningful error</li>
<li>Investigate why 3 seemingly identical errors are sent in separate emails around the same time</li>
</ul>
openQA Infrastructure - action #156226 (Resolved): [bot-ng] Pipeline failed / failed to pulled im...https://progress.opensuse.org/issues/1562262024-02-28T13:51:23Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2325569" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2325569</a></p>
<pre><code>WARNING: Failed to pull image with policy "always": failed to register layer: open /var/cache/zypp/solv/@System/solv.idx: no space left on device (manager.go:237:16s)
ERROR: Job failed: failed to pull image "registry.suse.de/qa/maintenance/containers/qam-ci-leap:latest" with specified policies [always]: failed to register layer: open /var/cache/zypp/solv/@System/solv.idx: no space left on device (manager.go:237:16s)
WARNING: Failed to pull image with policy "always": failed to register layer: mkdir /var/cache/zypp/solv/obs_repository: no space left on device (manager.go:237:13s)
ERROR: Job failed: failed to pull image "registry.suse.de/qa/maintenance/containers/qam-ci-leap:latest" with specified policies [always]: failed to register layer: mkdir /var/cache/zypp/solv/obs_repository: no space left on device (manager.go:237:13s)
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>DONE</strong> Restart pipelines</li>
<li><strong>DONE</strong> Report an infra SD ticket</li>
<li><strong>DONE</strong> Add retries to the pipeline</li>
</ul>
QA - action #156175 (Resolved): Support development of https://github.com/openSUSE/qem-bot/pull/1...https://progress.opensuse.org/issues/1561752024-02-27T19:07:49Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>QE Core and others would like to see improvements to qem-bot. So that we can enable them to develop new features themselves we should support. In this case <a href="https://github.com/openSUSE/qem-bot/pull/154" class="external">https://github.com/openSUSE/qem-bot/pull/154</a> shows a new concept. While the PR is not mergeable yet we can still try out the functionality. For this <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/merge_requests/65" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/merge_requests/65</a> already exists which we should merge, monitor and most likely revert after we have enough feedback which we should write back in <a href="https://github.com/openSUSE/qem-bot/pull/154" class="external">https://github.com/openSUSE/qem-bot/pull/154</a>.<br>
For more context see <a href="https://suse.slack.com/archives/C02CANHLANP/p1709051006649099" class="external">https://suse.slack.com/archives/C02CANHLANP/p1709051006649099</a></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> It is known if <a href="https://github.com/openSUSE/qem-bot/pull/154" class="external">https://github.com/openSUSE/qem-bot/pull/154</a> can generally work</li>
<li><strong>AC2:</strong> CI jobs in <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/</a> are green (again)</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Merge <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/merge_requests/65" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/merge_requests/65</a> when you are able to also monitor the impact</li>
<li>Monitor and revert as necessary</li>
<li>Feed feedback back to <a href="https://github.com/openSUSE/qem-bot/pull/154" class="external">https://github.com/openSUSE/qem-bot/pull/154</a></li>
<li>Rinse and repeat</li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>Have <a href="https://github.com/openSUSE/qem-bot/pull/154" class="external">https://github.com/openSUSE/qem-bot/pull/154</a> merged</li>
</ul>
QA - action #154498 (Resolved): [spike][timeboxed:20h][integration] Approve/reject SLE maintenanc...https://progress.opensuse.org/issues/1544982024-01-29T17:33:42Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>One of the most important responsibilities within SLE maintenance testing is to approve/reject SLE maintenance release requests based on openQA test results. So far <a href="https://github.com/openSUSE/qem-bot" class="external">qem-bot</a> is sufficient to schedule openQA tests but merely does a mediocre job of reporting back results as test results are asynchronously polled based on a periodic schedule <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules</a> causing unnecessary delays, inefficient polling, using outdated results <a class="issue tracker-4 status-4 priority-4 priority-default child" title="action: Use live openQA test results instead of inconsistent qem-dashboard database in qem-bot approver (Feedback)" href="https://progress.opensuse.org/issues/122311">#122311</a> and not even reporting back on blocking test failures <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121">#97121</a>. Let's use a proper architecture with efficient event based triggers providing relevant information back to release requests on IBS using core openQA features rather than too much custom lacking downstream tooling: Develop a proof-of-concept of listening to yet-to-be designed "openQA product build testing finished" AMQP events and approve/reject the according release request.</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Research how common OBS checks are implemented, e.g. openQA staging test integration, legalreview, installcheck, etc. For this see <a href="https://github.com/openSUSE/opensuse-release-tools" class="external">https://github.com/openSUSE/opensuse-release-tools</a>
<ul>
<li>Read <a href="https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/rabbit-openqa.py" class="external">https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/rabbit-openqa.py</a></li>
<li>Read <a href="https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/rabbit-repoid.py" class="external">https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/rabbit-repoid.py</a></li>
</ul></li>
<li>Follow <a class="issue tracker-4 status-3 priority-3 priority-lowest closed child" title="action: Find "last build" of a product over API size:M (Resolved)" href="https://progress.opensuse.org/issues/152939">#152939</a> and add publishing for an AMQP event for when incident "foo" finishes testing in openQA. For finding all tests related to incident "foo" see <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655">#117655</a></li>
<li>Integrate both of the above either in a new standalone application or hack into <a href="https://github.com/openSUSE/qem-bot" class="external">https://github.com/openSUSE/qem-bot</a> – as part of a spike solution so do not be afraid to break any other use case – to approve/"reject" SLE maintenance release requests. If "reject" seems to be too severe then provide only "informational" feedback, e.g. as IBS comment or checker result.</li>
<li>Optionally consider to implement this as a openQA plugin, maybe that is simpler for some cases</li>
</ul>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Also related to <a class="issue tracker-4 status-4 priority-4 priority-default child" title="action: Use live openQA test results instead of inconsistent qem-dashboard database in qem-bot approver (Feedback)" href="https://progress.opensuse.org/issues/122311">#122311</a>, <a class="issue tracker-6 status-1 priority-3 priority-lowest" title="coordination: [saga][epic] Re-combined Maintenance QA tooling covering both SLE+openSUSE (New)" href="https://progress.opensuse.org/issues/123088">#123088</a>, <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121">#97121</a>, <a class="issue tracker-6 status-1 priority-4 priority-default parent behind-schedule" title="coordination: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, ... (New)" href="https://progress.opensuse.org/issues/99303">#99303</a>, <a class="issue tracker-4 status-3 priority-3 priority-lowest closed child" title="action: Find "last build" of a product over API size:M (Resolved)" href="https://progress.opensuse.org/issues/152939">#152939</a>, <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: [timeboxed:6h][spike solution] a single command line or openQA webUI search view to show all test... (Resolved)" href="https://progress.opensuse.org/issues/131279">#131279</a>, <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655">#117655</a></p>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>Where to run persistently</li>
</ul>
openQA Project - action #154261 (Resolved): [spike][timeboxed:20h] batch commenting on all openQA...https://progress.opensuse.org/issues/1542612024-01-25T19:24:03Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>As proposed by szarate from a personal discussion between okurz and szarate 2024-01-25: When reviewing SLE maintenance tests openQA test failures failing for the same reason can be encountered which one wants to comment or label with the same content. To make such feature easily accessible it should be discoverable over the webUI.</p>
<a name="Goals"></a>
<h2 >Goals<a href="#Goals" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>G1:</strong> openQA jobs as filtered on the webUI, e.g. from /tests or /tests/overview, can be commented with the same text in one action</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Consider filters like <a href="https://openqa.opensuse.org/tests/overview?arch=aarch64&flavor=&machine=&test=&modules=&module_re=&group_glob=&not_group_glob=&comment=&todo=1&distri=opensuse&groupid=1&version=Tumbleweed#" class="external">https://openqa.opensuse.org/tests/overview?arch=aarch64&flavor=&machine=&test=&modules=&module_re=&group_glob=¬_group_glob=&comment=&todo=1&distri=opensuse&groupid=1&version=Tumbleweed#</a></li>
<li>As we already have a filter box on /tests/overview it might be the easiest to add a batch-comment-field in there. Then write a comment on all jobs currently shown in the filtered list (make it obvious that it is not a comment search field though)</li>
<li>As alternative consider something like selection boxes with possibility to select all, unselect all, select/unselect individuals</li>
<li>Consider a confirmation dialog if more than N jobs would be commented</li>
<li>Consider the audit log and a specific event for the batch action</li>
<li>See related issues <a class="issue tracker-4 status-1 priority-4 priority-default" title="action: Provide job settings filter options (New)" href="https://progress.opensuse.org/issues/120118">#120118</a> and <a class="issue tracker-6 status-1 priority-4 priority-default overdue child parent" title="coordination: [epic] Ideas from SUSE QE Tools workshop 2024-01-19 Filter openQA todo-jobs on /tests belonging t... (New)" href="https://progress.opensuse.org/issues/153928">#153928</a></li>
</ul>
QA - action #153682 (Resolved): Move of selected LSG QE machines NUE1 to PRG2e - quinn size:Mhttps://progress.opensuse.org/issues/1536822024-01-16T19:29:20Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> quinn is usable from PRG2e, i.e. as bare-metal test machine in OSD</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Follow <a href="https://jira.suse.com/browse/ENGINFRA-3737" class="external">https://jira.suse.com/browse/ENGINFRA-3737</a></li>
<li>Follow what has been done in <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Move of selected LSG QE machines NUE1 to PRG2e - fozzie size:M (Resolved)" href="https://progress.opensuse.org/issues/153670">#153670</a></li>
<li>Ensure PXE boot works, possibly same as #155521, simply add next_server config in OPS-Service/salt</li>
<li>Update openQA salt pillars</li>
<li>Ensure machine can be reached</li>
<li>Ensure openQA bare-metal tests using the machine are successful</li>
</ul>
QA - action #153670 (Resolved): Move of selected LSG QE machines NUE1 to PRG2e - fozzie size:Mhttps://progress.opensuse.org/issues/1536702024-01-16T19:21:24Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> fozzie is usable from PRG2e, i.e. as bare-metal test machine in OSD</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Follow <a href="https://jira.suse.com/browse/ENGINFRA-3734" class="external">https://jira.suse.com/browse/ENGINFRA-3734</a></li>
<li>Ensure PXE boot works, possibly same as #155521, simply add next_server config in OPS-Service/salt</li>
<li>Update openQA salt pillars</li>
<li>Ensure machine can be reached</li>
<li>Ensure openQA bare-metal tests using the machine are successful</li>
</ul>
openQA Project - action #135035 (Resolved): Optionally restrict multimachine jobs to a single workerhttps://progress.opensuse.org/issues/1350352023-09-01T11:38:16Zapappas
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Multi-machine jobs have been failing since 20230814, because of a misconfiguration of the MTU/GRE tunnels. A workaround has been found in forcing the complete multi-machine tests to run in the same worker.</p>
<p>The purpose of this ticket is to have all multi-machine runs be scheduled on the same well-configured worker.</p>
<p>The change doesn't need to be permanent but it does need to be applied until proper networking between multi-machine nodes can be guaranteed.</p>
<a name="Acceptance-Criteria"></a>
<h2 >Acceptance Criteria<a href="#Acceptance-Criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> If configured accordingly all jobs of a multi-machine parallel cluster must be scheduled to run on the <em>same</em> worker host</li>
<li><strong>AC2:</strong> By default jobs of a multi-machine parallel cluster can still be scheduled covering multiple different hosts</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Have a look at <a href="https://github.com/Martchus/openQA/pull/new/dependency-pinning" class="external">https://github.com/Martchus/openQA/pull/new/dependency-pinning</a> for how this could be enabled and documented.</li>
</ul>