openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-06T15:19:43ZopenSUSE Project Management Tool
Redmine QA - action #156775 (Resolved): cpanspec should adopt new %patch syntax size:Shttps://progress.opensuse.org/issues/1567752024-03-06T15:19:43Ztinitatina.mueller+trick-redmine@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a href="https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/YNXIWWHY7E2ZDMLKL44K7RR4Y2LCDV45/" class="external">https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/YNXIWWHY7E2ZDMLKL44K7RR4Y2LCDV45/</a></p>
<p><code>%patchN</code> is deprecated and should be replaced with <code>%patch -PN</code>.<br>
In cpanspec we are using <code>%autosetup</code> if possible, but otherwise we still use <code>%patchN</code>.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Perl packages in Factory (especially openQA ones) continue to build after the RPM change</li>
</ul>
<a name="Acceptance-tests"></a>
<h2 >Acceptance tests<a href="#Acceptance-tests" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AT1-1:</strong> Given RPM 4.20 is available all packages in devel:languages:perl still build without an error related to (at least) <code>%patchN</code></li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Check all packages in devel:languages:perl to ensure that they don't use <code>%patchN</code></li>
<li>Look into
<a href="https://github.com/openSUSE/cpanspec/blob/master/cpanspec#L1356-L1358" class="external">https://github.com/openSUSE/cpanspec/blob/master/cpanspec#L1356-L1358</a>
and adapt/extend to use explicit <code>%patch -PN</code> instead</li>
<li>Ensure all packages devel:languages:perl use that, e.g. with individual batch creation of submit requests. Possibly most are done already or don't rely on <code>%patch</code></li>
<li>Ensure that all submit requests are accepted and also forwarded into openSUSE:Factory accordingly</li>
<li>Crosscheck with scripting that no packages are left in d:l:p (and possibly openSUSE:Factory) using <code>%patchN</code></li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>Ignore devel:languages:perl:CPAN-[A-Z] as they would be only updated on new releases on CPAN but we don't care because those are not accepted (yet) into devel:languages:perl</li>
</ul>
openQA Project - action #156754 (Resolved): "DBIx::Class::Row::update(): Can't update OpenQA::Sch...https://progress.opensuse.org/issues/1567542024-03-06T11:43:54Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>As seen in <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: [alert] "HTTP Response" alert fired shortly on 2024-02-12 and 2024-03-04 size:M (Resolved)" href="https://progress.opensuse.org/issues/155326">#155326</a></p>
<p>OSD journal logs show some DBIx error:</p>
<pre><code>Feb 12 00:38:12 openqa openqa[11635]: [error] [ztQJ1_pAsMiS] DBIx::Class::Row::update(): Can't update OpenQA::Schema::Result::JobLocks=HASH(0x55b77ea45e28): row not found at /usr/share/openqa/script/../lib/OpenQA/Resource/Locks.pm line 139
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Specifically look into the only error in the log excerpt "[ztQJ1_pAsMiS] DBIx::Class::Row::update(): Can't update OpenQA::Schema::Result::JobLocks=HASH(0x55b77ea45e28): row not found at /usr/share/openqa/script/../lib/OpenQA/Resource/Locks.pm line 139"</li>
</ul>
openQA Infrastructure - action #156331 (Resolved): [gitlab] New pipeline schedules cannot be crea...https://progress.opensuse.org/issues/1563312024-02-29T12:50:10Zjbaier_czjbaier@suse.cz
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>New pipeline schedules can’t be created.</p>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<ol>
<li>Visit pipeline schedules of any project with CI/CD enabled.</li>
<li>Observe message: You have exceeded the maximum number of pipeline schedules for your plan. To create a new schedule, either increase your plan limit or delete an exisiting schedule.</li>
<li>See disabled button “New schedule”.</li>
</ol>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>New pipeline schedules can be created.</p>
<a name="Impact"></a>
<h2 >Impact<a href="#Impact" class="wiki-anchor">¶</a></h2>
<p>Without the ability to create more schedules, the automation process might be hindered.</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>This issue can be easily solved by following the steps mentioned in <a href="https://gitlab.suse.de/help/administration/instance_limits#number-of-pipeline-schedules" class="external">https://gitlab.suse.de/help/administration/instance_limits#number-of-pipeline-schedules</a></p>
QA - action #155917 (Resolved): [backlogger] Count "Feedback" ticket state for cycle time as well...https://progress.opensuse.org/issues/1559172024-02-23T10:37:06Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>As discussed in the coordination call where we now look at our metrics, it would be a good idea to take Feedback into account for the cycle time.</p>
<a name="Acceptance-Criteria"></a>
<h2 >Acceptance Criteria<a href="#Acceptance-Criteria" class="wiki-anchor">¶</a></h2>
<p><strong>AC1</strong>: The cycle time treats "Feedback" like "In Progress"<br>
<strong>AC2</strong>: We are aware that "Blocked" needs to be used (instead of "Feedback") when waiting on external progress</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Count "Feedback" the same as "In Progress". We include tickets in "Feedback" when just waiting for others within the team would be counted towards cycle time </li>
<li>Document in the wiki that waiting for external feedback should use Blocked</li>
</ul>
QA - action #154453 (Resolved): Move of selected LSG QE machines NUE1 to PRG2e - openqaw8-vmwarehttps://progress.opensuse.org/issues/1544532024-01-29T11:50:24Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> openqaw8-vmware is usable from PRG2e</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Follow <a href="https://jira.suse.com/browse/ENGINFRA-3841" class="external">https://jira.suse.com/browse/ENGINFRA-3841</a></li>
<li>Update openQA salt pillars</li>
<li>Ensure machine can be reached</li>
<li>Ensure openQA tests using the machine are successful</li>
</ul>
QA - action #154450 (Resolved): Move of selected LSG QE machines NUE1 to PRG2e - openqaw7-hypervhttps://progress.opensuse.org/issues/1544502024-01-29T11:48:05Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> openqaw7-hyperv is usable from PRG2e</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Follow <a href="https://jira.suse.com/browse/ENGINFRA-3840" class="external">https://jira.suse.com/browse/ENGINFRA-3840</a></li>
<li>Update openQA salt pillars</li>
<li>Ensure machine can be reached</li>
<li>Ensure openQA tests using the machine are successful</li>
</ul>
QA - action #154447 (Resolved): Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - gollumhttps://progress.opensuse.org/issues/1544472024-01-29T11:44:43Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> gollum is usable from PRG2</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Follow <a href="https://jira.suse.com/browse/ENGINFRA-3839" class="external">https://jira.suse.com/browse/ENGINFRA-3839</a></li>
<li>Ensure machine can be reached</li>
<li>Ensure machine is used as in before migration</li>
</ul>
openQA Infrastructure - action #152095 (Resolved): [spike solution][timeboxed:8h] Ping over GRE t...https://progress.opensuse.org/issues/1520952023-12-05T13:22:56Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>See lessons learned meeting <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Conduct "lessons learned" with Five Why analysis for "test fails in iscsi_client due to salt 'hos... (Resolved)" href="https://progress.opensuse.org/issues/139136">#139136</a>. We would again benefit from an easier reproducer. Related to <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: [kernel] minimal reproducer for many multi-machine test failures in "ovs-client+ovs-server" test ... (Resolved)" href="https://progress.opensuse.org/issues/135818">#135818</a> . Come up with a way to ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> We know how to ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><p>Research upstream about pinging over specific interfaces, GRE tunnels, TAP devices, openvswitch, etc.</p>
<ul>
<li>Like <code>ping -I<interface></code> or <code>ping X.X.X.X%tap0</code>?</li>
<li>Checkout network namespaces and if they could be used</li>
</ul></li>
<li><p>Research about MTU size debugging, tracepath, traceroute, etc.</p></li>
<li><p>Experiment in an openQA-environment or openQA-like with the bridges, tap devices, etc.</p></li>
<li><p>Demonstrate to the team in written form or interactively</p></li>
<li><p>Lookup how the existing check is done via a VM/VNC, and see how this could be simplified</p></li>
</ul>
openQA Infrastructure - action #150938 (Resolved): [openQA][sut][ipmi] No ipmi sol output with ix...https://progress.opensuse.org/issues/1509382023-11-16T09:39:37Zwaynechen55wchen@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Test run starts failing with <code>imagetester:7</code> at ipxe_install, for example, <a href="https://openqa.suse.de/tests/12822901#step/ipxe_install/1" class="external">https://openqa.suse.de/tests/12822901#step/ipxe_install/1</a>. It looks like needle matching failure, but actually there is nothing printed out on its ipmi sol console after reboot. </p>
<pre><code>ipmitool -I lanplus -C 3 -H ix64ph1075-sp.qe.nue2.suse.org -U admin -P xxxxxxxx sol activate
</code></pre>
<a name="Steps-to-reproduce"></a>
<h2 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h2>
<ul>
<li>Connect to ix64ph1075 ipmi sol console</li>
<li>Reboot the machine</li>
<li>Wait for output on ipmi sol console</li>
</ul>
<a name="Impact"></a>
<h2 >Impact<a href="#Impact" class="wiki-anchor">¶</a></h2>
<p>No test run assigned to <code>imagetester:7</code> can proceed. Now <code>imagetester:6</code></p>
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<ul>
<li>Looks like something wrong with ipmi sol console</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Check ipmi sol config</li>
<li>Check warning/error in BMC</li>
<li>Factory-reset the BMC</li>
<li>Reinstall the firmware</li>
<li>Click every possible button</li>
<li>Check that the physical ethernet cable is not broken</li>
</ul>
<a name="Workaround"></a>
<h2 >Workaround<a href="#Workaround" class="wiki-anchor">¶</a></h2>
<p>n/a</p>
<a name="Rollback-actions"></a>
<h2 >Rollback actions<a href="#Rollback-actions" class="wiki-anchor">¶</a></h2>
<ul>
<li><code>sudo systemctl unmask openqa-worker-auto-restart@6 && sudo systemctl enable --now openqa-worker-auto-restart@6</code></li>
</ul>
openQA Infrastructure - action #137600 (Resolved): [alert] Packet loss between worker hosts and o...https://progress.opensuse.org/issues/1376002023-10-09T07:46:02Zjbaier_czjbaier@suse.cz
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>We had multiple occurrences of packet loss alert over the weekend</p>
<pre><code>alertname Packet loss between worker hosts and other hosts alert
grafana_folder Salt
rule_uid 2Z025iB4km
</code></pre>
<p><a href="http://stats.openqa-monitor.qa.suse.de/d/EML0bpuGk?orgId=1&viewPanel=4" class="external">http://stats.openqa-monitor.qa.suse.de/d/EML0bpuGk?orgId=1&viewPanel=4</a></p>
<p>Currently, the problematic ones according to the panel are:</p>
<pre><code>imagetester - walter1.qe.nue2.suse.org 100%
petrol-1 - walter1.qe.nue2.suse.org 100%
sapworker1 - walter1.qe.nue2.suse.org 100%
</code></pre>
<p>That is a little bit weird as I manually checked the first one and it can reach each other well</p>
<pre><code>walter1:~ # ping imagetester.qe.nue2.suse.org
PING imagetester.qe.nue2.suse.org (10.168.192.249) 56(84) bytes of data.
64 bytes from imagetester.qe.nue2.suse.org (10.168.192.249): icmp_seq=7 ttl=64 time=0.326 ms
jbaier@imagetester:~> ping walter1.qe.nue2.suse.org
PING walter1.qe.nue2.suse.org (10.168.192.1) 56(84) bytes of data.
64 bytes from walter1.qe.nue2.suse.org (10.168.192.1): icmp_seq=1 ttl=64 time=0.331 ms
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Confirm <strong>when</strong> this started happening or if it's no longer an issue</li>
<li>There's no paused alerts</li>
</ul>
openQA Infrastructure - action #132860 (Resolved): openqa-piworker is unstable and needs regular ...https://progress.opensuse.org/issues/1328602023-07-17T08:39:49Zosukup
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1694765" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1694765</a></p>
<p>only thing found in logs:<br>
salt_ping.log:</p>
<pre><code>Currently the following minions are down:
8d7
< "openqa-piworker.qa.suse.de"
===================
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> we are able to process openQA Raspberry Pi bare-metal jobs consistently over some days</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><p>Identify the cause for regression</p>
<ul>
<li>likely something related to the hardware RTC</li>
<li>try if it just works with Leap 15.5 because we wanted to upgrade anyway</li>
<li>could be a recent kernel update so try to downgrade</li>
</ul></li>
<li><p>If it is really necessary and you exhausted all other remote-controllable options then go to the office, unplug RTC, reinstall the system assuming it was a borked system and corruption, or whatever</p></li>
<li><p>As Plan Y (if options A to X failed) buy wifi&bluetooth adapter for a IPMI controllable server and use that instead to connect to the rpi bare metal test instances</p></li>
</ul>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li>Add back salt key with <code>ssh osd "sudo salt-key -y -a openqa-piworker.qa.suse.de"</code></li>
</ul>
openQA Infrastructure - action #65178 (Resolved): Drop rsync.pl config from salt for osd and o3https://progress.opensuse.org/issues/651782020-04-02T10:15:17Zlivdywanliv.dywan@suse.com
<p>okurz wrote:</p>
<blockquote>
<p>oops, found that we have the repo still installed and configured for both osd and o3 which we should remove before we can call this done. E.g. see <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls#L177" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/server.sls#L177</a> . IMHO as long as we have the repo checked out and covered in salt we are not done with the cleanup.</p>
</blockquote>
openQA Infrastructure - action #44612 (Resolved): Do we want to update http://tumblesle.qa.suse.d...https://progress.opensuse.org/issues/446122018-12-01T07:30:43Zokurzokurz@suse.com
<p><a href="http://tumblesle.qa.suse.de/" class="external">http://tumblesle.qa.suse.de/</a></p>
openQA Infrastructure - action #37644 (Resolved): [tools] osd SSL certificate is only valid for o...https://progress.opensuse.org/issues/376442018-06-21T18:58:28Zokurzokurz@suse.comopenQA Infrastructure - action #19190 (Resolved): make use of ix64ph1014, e.g. for proxymodehttps://progress.opensuse.org/issues/191902017-05-17T07:11:33Zokurzokurz@suse.com
<p>[17 May 2017 08:52:46] coolo: do we still need <a href="https://openqa.suse.de/tests/latest?machine=ix64ph1014" class="external">https://openqa.suse.de/tests/latest?machine=ix64ph1014</a> ?<br>
[17 May 2017 08:54:40] okurz: well, I have no love left for this vnc thingie<br>
[17 May 2017 08:54:52] okurz: we better free this machine and use it for proxymode<br>
[17 May 2017 09:08:38] good morning<br>
[17 May 2017 09:09:53] coolo: so I will delete the machine in openQA and delete the schedule?<br>
[17 May 2017 09:10:51] okurz: leave the machine as documentation how to set this up. It might still be wanted in the future - for another machine<br>
[17 May 2017 09:11:00] but drop the job</p>
<p>okurz: I dropped the job from scheduling</p>