openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-07-18T07:56:34ZopenSUSE Project Management Tool
Redmine openQA Infrastructure - action #132926 (Workable): OSD cron -> (fetch_openqa_bugs)> /tmp/fetch_op...https://progress.opensuse.org/issues/1329262023-07-18T07:56:34Zosukup
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>OSD cron -> (fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log failed:</p>
<p>from traceback:</p>
<pre><code>requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/SUSE/ha-sap-terraform-deployments/issues/857 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f7439e43b38>, 'Connection to api.github.com timed out. (connect timeout=10)'))
</code></pre>
<p>fetch_openqa_bug failed when fetch issues from GitHub</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> It is understood why the error occurred</li>
<li><strong>AC2:</strong> The error does not persist</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Make sure you can login, see <a href="https://gitlab.suse.de/OPS-Service/salt/-/blob/production/pillar/id/openqa-service_qe_suse_de.sls#L11" class="external">https://gitlab.suse.de/OPS-Service/salt/-/blob/production/pillar/id/openqa-service_qe_suse_de.sls#L11</a> or ask dheidler/mkittler to do that for you</li>
<li>Assuming "host unavailable', check how long the scripts retried
<ul>
<li>Re-try more often?</li>
<li>Wait longer between attemps? </li>
</ul></li>
<li><a href="https://github.com/os-autoinst/openqa_bugfetcher" class="external">https://github.com/os-autoinst/openqa_bugfetcher</a></li>
</ul>
QA - action #126953 (Resolved): [tools] mtui hangs on disconnect if network is offlinehttps://progress.opensuse.org/issues/1269532023-03-30T09:47:18Zosukup
<p>see - <a href="https://github.com/openSUSE/mtui/issues/11" class="external">https://github.com/openSUSE/mtui/issues/11</a></p>
QA - action #123367 (Resolved): [tools] drop old testplatform code from teregen, repose and produ...https://progress.opensuse.org/issues/1233672023-01-19T10:08:27Zosukup
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>We finally have all refhosts in metadata described in the new format so we can safely drop old testplatform.</p>
<p>current situation: Teregen generates testplatform in two different formats based on two different product definitions which leads to data duplication and confuse users which data file need to be updated for correctly working template generation. Repose can generate two different yaml formats, it was changed to defaults in new but user still can post old format, so removing this possibility will reduce user related errors when maintaining refhosts files.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> No old testplatform generated by teregen</li>
<li><strong>AC2:</strong> productsdef.pm file dropped from metadata</li>
<li><strong>AC3:</strong> old yaml format removed from repose</li>
</ul>
<a name="Links"></a>
<h2 >Links<a href="#Links" class="wiki-anchor">¶</a></h2>
<ul>
<li><a href="https://gitlab.suse.de/qa-maintenance/teregen" class="external">https://gitlab.suse.de/qa-maintenance/teregen</a></li>
<li><a href="https://gitlab.suse.de/qa-maintenance/metadata/-/blob/master/productdefs.pm" class="external">https://gitlab.suse.de/qa-maintenance/metadata/-/blob/master/productdefs.pm</a></li>
<li><a href="https://github.com/openSUSE/repose" class="external">https://github.com/openSUSE/repose</a></li>
<li><a href="https://confluence.suse.com/display/maintenanceqa/Repose+Usage" class="external">https://confluence.suse.com/display/maintenanceqa/Repose+Usage</a></li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Disable relevant module in teregen template generator</li>
<li>Drop <code>--yaml-old</code> option in repose (keep <code>--yaml-ng</code>)</li>
</ul>
QA - action #113087 (Resolved): [qa-tools][qem-bot] malformed data in smelt incident causes smelt...https://progress.opensuse.org/issues/1130872022-06-27T16:51:08Zosukup
<p><a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1032830" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1032830</a></p>
<pre><code>ERROR: Expecting value: line 1 column 1 (char 0)
8889Traceback (most recent call last):
8890 File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/loader/smelt.py", line 64, in get_incident
8891 inc_result = requests.get(SMELT, params={"query": query}, verify=False).json()
8892 File "/usr/lib/python3.6/site-packages/requests/models.py", line 898, in json
8893 return complexjson.loads(self.text, **kwargs)
8894 File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 525, in loads
8895 return _default_decoder.decode(s)
8896 File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
8897 obj, end = self.raw_decode(s)
8898 File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
8899 return self.scan_once(s, idx=_w(s, idx).end())
8900simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
</code></pre> QA - action #112430 (Resolved): [qa-tools] [qem-bot] Incident schedule fails in preparation Incid...https://progress.opensuse.org/issues/1124302022-06-14T17:10:56Zosukup
<p>from log:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="n">INFO</span><span class="p">:</span> <span class="mi">2022</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">14</span> <span class="mi">17</span><span class="p">:</span><span class="mi">03</span><span class="p">:</span><span class="mf">40.405480</span><span class="p">:</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">download</span><span class="p">.</span><span class="n">suse</span><span class="p">.</span><span class="n">de</span><span class="o">/</span><span class="n">ibs</span><span class="o">/</span><span class="n">SUSE</span><span class="p">:</span><span class="o">/</span><span class="n">Maintenance</span><span class="p">:</span><span class="o">/</span><span class="mi">24419</span><span class="o">/</span><span class="n">SUSE_Updates_SLE</span><span class="o">-</span><span class="n">WE_12</span><span class="o">-</span><span class="n">SP5_x86_64</span><span class="o">/</span><span class="n">repodata</span><span class="o">/</span><span class="n">repomd</span><span class="p">.</span><span class="n">xml</span> <span class="ow">not</span> <span class="n">found</span> <span class="o">--</span> <span class="n">skip</span> <span class="n">incident</span>
<span class="n">INFO</span><span class="p">:</span> <span class="n">Project</span> <span class="n">SUSE</span><span class="p">:</span><span class="n">Maintenance</span><span class="p">:</span><span class="mi">24419</span> <span class="n">can</span><span class="sh">'</span><span class="s">t calculate repohash .. skipping
INFO: Project SUSE:Maintenance:24462 has empty channels - check incident in SMELT
Traceback (most recent call last):
File </span><span class="sh">"</span><span class="s">./qem-bot/bot-ng.py</span><span class="sh">"</span><span class="s">, line 7, in <module>
main()
File </span><span class="sh">"</span><span class="s">/builds/qa-maintenance/bot-ng/qem-bot/openqabot/main.py</span><span class="sh">"</span><span class="s">, line 41, in main
sys.exit(cfg.func(cfg))
File </span><span class="sh">"</span><span class="s">/builds/qa-maintenance/bot-ng/qem-bot/openqabot/args.py</span><span class="sh">"</span><span class="s">, line 24, in do_incident_schedule
bot = OpenQABot(args)
File </span><span class="sh">"</span><span class="s">/builds/qa-maintenance/bot-ng/qem-bot/openqabot/openqabot.py</span><span class="sh">"</span><span class="s">, line 23, in __init__
self.incidents = get_incidents(self.token)
File </span><span class="sh">"</span><span class="s">/builds/qa-maintenance/bot-ng/qem-bot/openqabot/loader/qem.py</span><span class="sh">"</span><span class="s">, line 41, in get_incidents
xs.append(Incident(i))
File </span><span class="sh">"</span><span class="s">/builds/qa-maintenance/bot-ng/qem-bot/openqabot/types/incident.py</span><span class="sh">"</span><span class="s">, line 27, in __init__
for r in incident[</span><span class="sh">"</span><span class="s">channels</span><span class="sh">"</span><span class="s">]
File </span><span class="sh">"</span><span class="s">/builds/qa-maintenance/bot-ng/qem-bot/openqabot/types/incident.py</span><span class="sh">"</span><span class="s">, line 25, in <listcomp>
for p, v, a in (
ValueError: not enough values to unpack (expected 3, got 2)
</span></code></pre> QA - action #109977 (Resolved): qem-bot - approve pipeline failed with 403 forbidden size:Mhttps://progress.opensuse.org/issues/1099772022-04-14T10:45:36Zosukup
<p>Gitlab CI <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/929031">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/929031</a></p>
<p>after look to IBS , RR (release request) was accepted before but info about this didn't propagate to dashboard (some latency is excepted IBS->SMELT->Dashboard), so new try to accept accepted RR ends with error</p>
<pre><code>INFO: Accepting review for SUSE:Maintenace:23688:269540
ERROR: HTTP Error 403: Forbidden
Traceback (most recent call last):
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/approver.py", line 117, in osc_approve
message=msg,
File "/usr/lib/python3.6/site-packages/osc/core.py", line 4323, in change_review_state
f = http_POST(u, data=message)
File "/usr/lib/python3.6/site-packages/osc/core.py", line 3422, in http_POST
def http_POST(*args, **kwargs): return http_request('POST', *args, **kwargs)
File "/usr/lib/python3.6/site-packages/osc/core.py", line 3410, in http_request
fd = urlopen(req, data=data)
File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python3.6/urllib/request.py", line 564, in error
result = self._call_chain(*args)
File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib64/python3.6/urllib/request.py", line 1059, in http_error_401
url, req, headers)
File "/usr/lib64/python3.6/urllib/request.py", line 1007, in http_error_auth_reqed
return self.retry_http_basic_auth(host, req, realm)
File "/usr/lib64/python3.6/urllib/request.py", line 1022, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
</code></pre>
<a name="Acceptance-criteria"></a>
<h1 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h1>
<ul>
<li><strong>AC1:</strong> Approve runs without error in case RR is already processed</li>
</ul>
<a name="Suggestions"></a>
<h1 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h1>
<ul>
<li>Check response before throwing exception and act accordingly</li>
<li>Check IBS RR if really has open review for <code>qam-openqa</code></li>
<li>The meaning of the 403 response code here is that we try to accept something that is already accepted. How about catching the case where the XML indicates the specific error, logging and ignoring the case.</li>
<li>Log server response (not just return code)</li>
<li>Suggest a better response (and response code) to OBS upstream</li>
</ul>
QA - action #109878 (Resolved): bot-ng schedule/approve abortedhttps://progress.opensuse.org/issues/1098782022-04-12T16:18:40Zosukup
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>postgresql container on qam2.suse.de changed again IP and after this dashboard lost connection with database</p>
<p>after restart of dashboard service ( and reconnect to database using new ip) service for some time worked, but sometimes connection between container and host seems broken.</p>
<pre><code>-- Logs begin at Tue 2022-04-12 18:07:21 CEST. --
dub 12 18:07:26 qam2 dnsmasq-dhcp[616]: DHCPREQUEST(br0) 192.168.0.48 aa:5d:de:99:79:13
dub 12 18:07:26 qam2 dnsmasq-dhcp[616]: DHCPACK(br0) 192.168.0.48 aa:5d:de:99:79:13 postgresql
dub 12 18:13:27 qam2 dnsmasq-dhcp[616]: DHCPDISCOVER(br0) aa:5d:de:99:79:13
dub 12 18:13:27 qam2 dnsmasq-dhcp[616]: DHCPOFFER(br0) 192.168.0.168 aa:5d:de:99:79:13
dub 12 18:13:27 qam2 dnsmasq-dhcp[616]: DHCPREQUEST(br0) 192.168.0.48 aa:5d:de:99:79:13
dub 12 18:13:27 qam2 dnsmasq-dhcp[616]: DHCPACK(br0) 192.168.0.48 aa:5d:de:99:79:13 postgresql
dub 12 18:13:37 qam2 dnsmasq-dhcp[616]: DHCPDISCOVER(br0) aa:5d:de:99:79:13
dub 12 18:13:37 qam2 dnsmasq-dhcp[616]: DHCPOFFER(br0) 192.168.0.168 aa:5d:de:99:79:13
dub 12 18:13:37 qam2 dnsmasq-dhcp[616]: DHCPREQUEST(br0) 192.168.0.48 aa:5d:de:99:79:13
dub 12 18:13:37 qam2 dnsmasq-dhcp[616]: DHCPACK(br0) 192.168.0.48 aa:5d:de:99:79:13 postgresql
</code></pre>
<a name="Acceptance-crtieria"></a>
<h2 >Acceptance crtieria<a href="#Acceptance-crtieria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Pipeline doesn't fail due dashboard connection problems with db</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>implement <a href="https://progress.opensuse.org/issues/106547" class="external">https://progress.opensuse.org/issues/106547</a></li>
</ul>
QA - action #109512 (Resolved): qem-bot - add vars with GitlabCI job link and qem-dashboard linkhttps://progress.opensuse.org/issues/1095122022-04-05T18:11:21Zosukup
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>made easy to find which Gitlab CI job scheduled incident / aggregate in openQA </p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> added variables with clickable URLs to Schedule and QEM dashboard to job variables</li>
</ul>
QA - action #109509 (New): qem-dashboard - show better info about time of actualization of datahttps://progress.opensuse.org/issues/1095092022-04-05T18:07:39Zosukup
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>We already have a "last updated" in the dashboard which can be misleading. </p>
<p>We should have - last time of SMELT sync, last time of Incidents schedule and last time of Aggregates Schedule</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Correct update datetime is shown on dashboard</li>
</ul>
QA - action #109491 (New): Flow diagram for Maintenance jobs scheduling https://progress.opensuse.org/issues/1094912022-04-05T13:26:40Zosukup
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>For better understanding how are Maintenance Incidents scheduled and workflow around incidents in QEM + openQA</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Flow diagram in our wiki or qem-bot/qem-dashboard docs</li>
</ul>
openQA Project - coordination #101382 (Workable): [openqa][epic] update of assets in openQA https://progress.opensuse.org/issues/1013822021-10-22T15:07:45ZosukupQA - action #97994 (New): bot-ng - sometimes doesn't update smelt data in dashboardhttps://progress.opensuse.org/issues/979942021-09-02T16:31:29Zosukup
<p>see <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/557971#L321" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/557971#L321</a></p>
<pre><code>ERROR: Smelt Incidents wern't synced to dashboard
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Incidents are always synced</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Make the error message useful</li>
<li>Investigate why we have error 400</li>
<li>Look at the dashboard side of it also</li>
</ul>
<a name="Work-around"></a>
<h2 >Work-around<a href="#Work-around" class="wiki-anchor">¶</a></h2>
<ul>
<li>We trust the next pipline runs cover retry sufficiently</li>
</ul>
openQA Project - action #94372 (New): [qem][tools] Use GitLab Pages to expose MTUI documentationhttps://progress.opensuse.org/issues/943722021-06-21T14:56:50Zosukup
<p>We now have docu on <a href="http://qam.suse.de/projects/mtui/latest" class="external">http://qam.suse.de/projects/mtui/latest</a> which needs +- 5 manual steps to deploy this and almost nobody knows this addres</p>
<p>--> GitlabPages actual documentation after any change<br>
--> gitlab generated url with perspective for future custom domains support :lloking_forward:</p>
QA - action #93937 (New): [tools][qem] qamapi - incidents can have multiple versions packageshttps://progress.opensuse.org/issues/939372021-06-14T08:52:32ZosukupopenQA Tests - action #48053 (New): [aarch64] test fails in system_workaroundshttps://progress.opensuse.org/issues/480532019-02-18T09:46:36Zosukup
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-15-Installer-DVD-QR-aarch64-qam-allpatterns+addons@aarch64 fails in<br>
<a href="https://openqa.suse.de/tests/2465249/modules/system_workarounds/steps/2" class="external">system_workarounds</a></p>
<p>looks like test try open nonexisting console on tty6..</p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/2433622" class="external">0009</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/2415056" class="external">0009</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?flavor=Installer-DVD-QR&test=qam-allpatterns%2Baddons&machine=aarch64&arch=aarch64&distri=sle&version=15" class="external">latest</a></p>