QA

I don't know if these are related. But it looks like osd is responsive. And it's only https://openqa.suse.de/tests/overview requests that seem to time out. It doesn't look like the network is the problem:

ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590756'. Could be deleted workaround needle?.
[...]
ERROR:openqa_review.openqa_review:Failed to process {'state': 'NEW_SOFT_ISSUE', 'href': '/tests/6230452', 'failedmodules': []} with error Request to https://openqa.suse.de/tests/6230452/file/details-console_reboot#1.json was not successful, status code: 404. Skipping current result

Maybe related to #92854 🤔️

"Could not find any soft failure" and the other error are not related, these are different problems.

For reference the command to reproduce locally:

mkdir -p
python3 ./openqa_review/openqa_review.py --host https://openqa.suse.de -n -r -T --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --save --save-dir out

(Side note, I could not avoid having valid bugzilla credentials even after dropping --query-issue-status, maybe an unrelated bug)

To reproduce the error I changed the host to test.openqa.suse.de (.invalid doesn't work here).

I want to say this is by definition not a 502/503/504 since the error would say "was not successful, status code: ". But the code should preserve at least the last exception or request status. Currently due to the retries that gets lost.

https://github.com/os-autoinst/openqa_review/pull/147

the format of your comment is broken.

(Side note, I could not avoid having valid bugzilla credentials even after dropping --query-issue-status, maybe an unrelated bug)

That's expected because of the option --reminder-comment-on-issues which I suggest you avoid in local testing with live instances as that means that your user account will write comments on bugzilla.

Thanks! Fixed the formatting (had some rogue line breaks in there) and dropped --reminder-comment-on-issues from the command.

https://github.com/os-autoinst/openqa_review/pull/147 merged. The gitlab CI pipeline uses container images using the released packages of openqa-review from openSUSE Tumbleweed. So after I would create a release and create a submit request to openSUSE Factory some days can pass until the gitlab CI pipeline has access to a more recent version.

How about using /tests/overview.json instead of HTML document parsing and the suggested additional retries?

Ack, I'm going to propose additional PRs. I had thought it was gonna go quicker. In hindsight I should've prepared those in parallel.

okurz wrote:

How about [...] the suggested additional retries?

https://github.com/os-autoinst/openqa_review/pull/148

cdywan wrote:

okurz wrote:

How about [...] the suggested additional retries?

https://github.com/os-autoinst/openqa_review/pull/148

After proposing the simple increase I also pondered a little more and decided to try and replace the loop and else approach:

https://github.com/os-autoinst/openqa_review/pull/149

This is similar to what's described here although I didn't create a new class for a single use: https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/

okurz wrote:

How about using /tests/overview.json instead of HTML document parsing[...]

The "soup" parsing is not very obvious at first, code like find_all(class_="badge") w/ no documentation of the expected data, so I had to do some reverse engineering. There's also no test coverage so I'm looking into that first. Iterating changes with real requests is way too slow.

For reference the relevant "soup" would look something like this:

    <div id="summary" class="card border-success">
        <div class="card-header">
            Overall Summary of
                <strong><a href="/group_overview/4">openSUSE Tumbleweed PowerPC</a></strong>
                build 20160917
            <div class="time-params">
                    showing latest jobs,
                    <a href="/tests/overview?distri=opensuse&amp;version=Tumbleweed&amp;build=20160917&amp;groupid=4&amp;t=2021-06-22+10%3A54%3A30+%2B0000">overview fixed to the current time</a>
            </div>
        </div>
        <div class="card-body">
            Passed: <span class="badge badge-success">0</span>
            Failed: <span class="badge badge-danger">0</span>
        </div>
    </div>

    <div class="card border-danger" id="summary">
<div class="card-header">Overall Summary of <strong><a href="/group_overview/377">Create_hdd_stable_hosts</a></strong> build 15_GM <div class="time-params"> showing latest jobs,  <a href="/tests/overview?distri=sle&amp;version=15&amp;build=15_GM&amp;groupid=377&amp;t=2021-06-21+16%3A23%3A57+%2B
000">overview fixed to the current time</a></div></div><div class="card-body">Passed: <span class="badge badge-success">0</span>Soft-Failed:<span class="badge badge-warning">2</span>Failed: <span class="badge badge-danger">1</span></div></div>
<i class="status fa fa-circle result_passed" title="Done: passed"></i>
</a>

<div class="card border-danger" id="summary">
<div class="card-header">Overall Summary of <strong><a href="/group_overview/377">Create_hdd_stable_hosts</a></strong>build 15-SP3_GM<div class="time-params">showing latest jobs,<a href="/tests/overview?distri=sle&amp;version=15-SP3&amp;build=15-SP3_GM&amp;groupid=377&amp;t=2021-06-21+16%3A26%3
A08+%2B0000">overview fixed to the current time</a></div></div><div class="card-body">Passed: <span class="badge badge-success">3</span>Incomplete:<span class="badge badge-secondary">1</span>Failed: <span class="badge badge-danger">0</span></div></div>
<i class="status fa fa-circle result_incomplete" title="Done: incomplete"></i>

<h3>Flavor: Installer-DVD-POST</h3>                                                                 
<table class="overview fixedheader table table-striped table-hover" id="results_Installer-DVD-POST">
<thead>                                                                                             
<tr>                                                                                                
<th>Test</th>                                                                                       
<th id="flavor_Installer-DVD-POST_arch_ppc64le">ppc64le</th>                                        
<th id="flavor_Installer-DVD-POST_arch_s390x">s390x</th>                                            
<th id="flavor_Installer-DVD-POST_arch_x86_64">x86_64</th>                                          
</tr>                         
</thead>                    
<!-- body omitted -->
</table>

Not sure, if that is related, but it seems that the overview page is still a problem sometimes, see https://gitlab.suse.de/openqa/openqa-review/-/jobs/474652

jbaier_cz wrote:

Not sure, if that is related, but it seems that the overview page is still a problem sometimes, see https://gitlab.suse.de/openqa/openqa-review/-/jobs/474652

Yes. My PR wasn't a magical fix but meant to make it more robust and reveal errors rather than ignoring them.

WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=2.5)")': /api/v1/jobs/5990475/details
[...]
WARNING:urllib3.connectionpool:Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=2.5)")': /tests/overview?distri=sle&version=15-SP3&build=186.1&groupid=143

So these are expected. I'm not sure what this one is, though. It looks like we're getting corrupt JSON. I'm thinking we should dump the corrupt data (which might be valid unexpected data). I've not seen this locally:

  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 996, in _get_bugref_for_softfailed_module
    details_json = self.test_browser.get_json(details_url)
  File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 75, in get_json
    return self.get_page(url, as_json=True, cache=cache)
  File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 102, in get_page
    content = self._get(absolute_url, as_json=as_json)
  File "/usr/lib/python3.8/site-packages/openqa_review/browser.py", line 147, in _get
    content = r.json() if as_json else r.content.decode("utf8")
[...]
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

https://github.com/os-autoinst/openqa_review/pull/162

I will probably state the obvious, judging from json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0), the content is not a JSON or (more probably) the content is just empty:

import json
>>> json.loads('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Related MRs:

• https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/6
• https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/7 And @okurz chaned the time of the pipeline

We discussed it after the weekly (okurz, cdywan, jbaier, dheidler) and applied the following changes:

• Changed the cron schedule in https://gitlab.suse.de/openqa/openqa-review/-/pipeline_schedules from 0 5 * * 1-5 to 47 23 * * * to
  • avoid full-hour congestion
  • avoid potentially busy time with work-time overlap from both APAC+EMEA
  • also run during less -congested weekend to crosscheck if the behaviour differs load-dependant
• Still generate the complete status page with the generated reports even if some fail: https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/6
• Retry steps in gitlab on failures: https://gitlab.suse.de/openqa/openqa-review/-/merge_requests/7
• Changed https://build.opensuse.org/projects/home:okurz:container:ca/meta to pull the python-openqa_review package from https://build.opensuse.org/package/show/home:okurz/python-openqa_review which is updated on every git commit to master in https://github.com/os-autoinst/openqa_review

EDIT: Then we see requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://progress.opensuse.org/issues/94342.json

https://gitlab.suse.de/openqa/openqa-review/-/settings/ci_cd has the redmine API key. With something like curl -s -H 'X-Redmine-API-Key: XXX' 'https://progress.opensuse.org/issues/94342.json' we can access private tickets so the API key seems to be fine. Maybe the implementation within openqa-review does not work there.

Added a PR for the progress API issue: https://github.com/os-autoinst/openqa_review/pull/164

@cdywan turns out that problems have actually been worsened with 60ebc73 from https://github.com/os-autoinst/openqa_review/pull/149 as found out by

git bisect start HEAD 1.19.0 
git bisect run python3 openqa_review/openqa_review.py --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old|EOL)' -J https://openqa.suse.de/group_overview/143

so we can revert multiple commits but likely the requests.retry approach is still good. But now we have a better reference with 1.19.0. In the current status with https://openqa.io.suse.de/openqa-review/ now showing basically a single report file this is becoming even more urgent now.

cdywan wrote:

okurz wrote:

problems have actually been worsened with 60ebc73 from https://github.com/os-autoinst/openqa_review/pull/149
[...]
so we can revert multiple commits but likely the requests.retry approach is still good

I don't follow. There's exactly one commit and that uses Retry instead of a loop.

Yes, I was also surprised but I believe what "git bisect" is telling us :) Please make sure that the team acts that with urgency, doesn't have to be you.

okurz wrote:

cdywan wrote:

okurz wrote:

problems have actually been worsened with 60ebc73 from https://github.com/os-autoinst/openqa_review/pull/149
[...]
so we can revert multiple commits but likely the requests.retry approach is still good

I don't follow. There's exactly one commit and that uses Retry instead of a loop.

Yes, I was also surprised but I believe what "git bisect" is telling us :) Please make sure that the team acts that with urgency, doesn't have to be you.

Ack. I'll propose a new PR handling network errors. So that in a report they should result in a message w/o stopping other reports being generated. This should bring us back to a state similar to the loop which swalloed errors. If not, let's decide tomorrow if we'd rather go back to an older version.

Good news, I managed to reproduce a HTTPError manually and proposed a PR to handle it: https://github.com/os-autoinst/openqa_review/pull/166

I think this should be "In Progress" as I understand you actively work on this.

As you seem to have problems with reproducing this, please monitor what actually happens in the pipelines: https://gitlab.suse.de/openqa/openqa-review/-/pipelines

For example:

• https://gitlab.suse.de/openqa/openqa-review/-/jobs/483641 shows "WARNING: Uploading artifacts as "archive" to coordinator... failed id=483641 responseStatus=502 Bad Gateway status=502", no one here seems to have mentioned this problem
• https://gitlab.suse.de/openqa/openqa-review/-/jobs/483642#L147 and four other jobs in the same pipeline show openqa_review.browser.DownloadError: Request to https://openqa.suse.de/tests/overview?distri=sle&version=15-SP3&build=187.1&groupid=110 was not successful after 7 retries: HTTPSConnectionPool(host='openqa.suse.de', port=443): Max retries exceeded with url: /tests/overview?distri=sle&version=15-SP3&build=187.1&groupid=110 (Caused by ReadTimeoutError("HTTPSConnectionPool(host='openqa.suse.de', port=443): Read timed out. (read timeout=2.5)")) or similar. Your PR does not handle that as it turns another error also into a DownloadError

I'm also testing with an increased timeout for individual requests, because I noticed that we end up retrying without ever succeeding if the server is too slow to respond faster: https://github.com/os-autoinst/openqa_review/pull/167

• https://gitlab.suse.de/openqa/openqa-review/-/jobs/483641 shows "WARNING: Uploading artifacts as "archive" to coordinator... failed id=483641 responseStatus=502 Bad Gateway status=502", no one here seems to have mentioned this problem

I've seen this but not sure what it means. The thing is, I can download the file artifacts.zip and it contains e.g. openqa_opensuse_org_status.html. So it doesn't seem like it fails in the end?

Also for further reference, I artifically tested that the report continues if all individual results fail. It will fail, though, if e.g. the overview route is unable to manage 10 seconds after 7 times bevause then we have no data at all. But individual ones don't matter.

Also, https://gitlab.suse.de/openqa/openqa-review/-/jobs/484379 looks really good again.

Yes, very good progress. Thank you! If the next "nightly" job is again problematic we can consider yet another point in time when the reports should be generated. Maybe now we are conflicting with some bigger schedule triggering from qa-maintenance/openQABot

okurz wrote:

Yes, very good progress. Thank you! If the next "nightly" job is again problematic we can consider yet another point in time when the reports should be generated. Maybe now we are conflicting with some bigger schedule triggering from qa-maintenance/openQABot

I retried several pipelines which ran at different times during the day. The puzzling "upload error" didn't come back so far. Do we consider this good enough?

cdywan wrote:

I retried several pipelines which ran at different times during the day. The puzzling "upload error" didn't come back so far. Do we consider this good enough?

Yes, because we should react on pipeline failures during our normal alert handling anyway