action #75214
closedopenqa-review fails to post reminder comments on bugzilla, errors in log "Encountered error trying to post a reminder comment on issue"
0%
Description
Observation¶
from https://gitlab.suse.de/openqa/openqa-review/-/jobs/275810/raw
++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.pZbTFYl7W0
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ERROR:openqa_review.openqa_review:Encountered error trying to post a reminder comment on issue '* [jeos-base+phub_hyperv@svirt-hyperv-uefi](https://openqa.suse.de/tests/4846363 "Failed modules: firstrun"), [jeos-filesystem_hyperv@svirt-hyperv-uefi](https://openqa.suse.de/tests/4846366 "Failed modules: bootloader_uefi") -> [bsc#1176134](https://bugzilla.suse.com/show_bug.cgi?id=1176134 "[Build 20.3] JeOS images 15sp3 do not boot in uefi mode") (Ticket status: <span style="color: red;">VERIFIED (FIXED)</span>, prio/severity: P2/Normal, assignee: ms@suse.com)
': 500 Server Error: Internal Server Error for url: https://apibugzilla.suse.com/jsonrpc.cgi. Skipping.
Steps to reproduce¶
- See the instructions called in the gitlab CI definitions in https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L44
Be aware that the gitlab CI pipeline uses a dedicated user account as visible in https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L9 and https://gitlab.suse.de/openqa/openqa-review/-/settings/ci_cd (see "Variables")
Problem¶
Posting reminder comments from account "okurz" works. This is what we use so far on lord.arch.suse.de. Also it worked to post a reminder comment manually in https://bugzilla.suse.com/show_bug.cgi?id=1005824#c10 but so far posting from gitlab CI did not work
Suggestions¶
- Try again manually with the credentials of the bot account
- Try to read/post bugs over API as openqa-review, e.g. in test bug https://bugzilla.suse.com/show_bug.cgi?id=1005824
- Crosscheck on https://bugzilla.suse.com/buglist.cgi?quicksearch=commenter%3Aopenqa-review if comments are posted correctly
Further details¶
Initially reported in #52655#note-10
Updated by okurz almost 4 years ago
- Related to action #52655: [epic] Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution added
Updated by Xiaojing_liu over 3 years ago
- Status changed from Workable to In Progress
- Assignee set to Xiaojing_liu
I would like to try this. Also it's a chance to learn Python.
Updated by openqa_review over 3 years ago
- Due date set to 2021-03-10
Setting due date based on mean cycle time of SUSE QE Tools
Updated by Xiaojing_liu over 3 years ago
I did a test using openqa_review
to get comments from https://bugzilla.suse.com/show_bug.cgi?id=1005824. it only can get the public comments. I tried this use my own account, all comments I could get. So I think openqa_review
does not have the permission. After I changed the is_private
to False
, the comment was added successfully. See: https://bugzilla.suse.com/show_bug.cgi?id=1005824 comments 11 and 13.
Not sure if we should upgrade openqa_review
permission, I created a pr to change the parameter: https://github.com/os-autoinst/openqa_review/pull/139. If we need to update the permission, I can close the pull request.
Updated by Xiaojing_liu over 3 years ago
I run openqa-review locally
python2.7 -m openqa_review.openqa_review --host http://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --job-groups '15 SP3' --reminder-comment-on-issues
After changing the is_private : False
, the comment was added to bugzilla successfully.
see: https://bugzilla.suse.com/show_bug.cgi?id=1179434 comment#27
But CI pipeline still report error 500: 500 Server Error: Internal Server Error for url
: https://gitlab.suse.de/openqa/openqa-review/-/jobs/349401/raw
Updated by Xiaojing_liu over 3 years ago
I tested in container:
liuxiaojing@linux-90dc:~> sudo podman run --rm -it registry.suse.de/home/okurz/container/images/openqa-review /bin/bash
According to gitlab log message:
[32;1m$ sh -ex bin/openqa-review-daily-email $ARGS[0;m
++ echo '[32;1m$ sh -ex bin/openqa-review-daily-email $ARGS[0;m'
++ sh -ex bin/openqa-review-daily-email
++ dirname bin/openqa-review-daily-email
+ . bin/_common
++ set -o pipefail
+ '[' -z ']'
+ setup_tmpdir
++ mktemp -d
+ tmp=/tmp/tmp.uXWMAvvkIL
+ trap cleanup EXIT
+ openqa_host=https://openqa.suse.de
+ recv=openqa-suse-status@suse.de
+ html_target_file=openqa_suse_de_status.html
+ html_target=openqa_suse_de_status.html
+ openqa_review_args='--host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups ^(Released|Development|old)'
+ load_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_email_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_html_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_save_args='--reminder-comment-on-issues --save --save-dir /tmp/tmp.uXWMAvvkIL'
++ command -v openqa-review
+ openqa_review=/usr/bin/openqa-review
++ dirname bin/openqa-review-daily-email
+ TPL=bin/../dashboard_files/dashboard.html.in
++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.uXWMAvvkIL
It runs /usr/bin/openqa-review ...
, this file belongs:
32c2adb65251:/openqa_review # rpm -qf /usr/bin/openqa-review
python38-openqa_review-1.20.2-1.3.noarch
this package doesn't include the pr https://github.com/os-autoinst/openqa_review/pull/139.
Not sure why we do git clone https://github.com/os-autoinst/openqa_review.git .
in gitlab, we don't use these codes.
Should we update the container image to let it includes the change (https://github.com/os-autoinst/openqa_review/pull/139)? or change the gitlab-ci to use the github code?
Updated by livdywan over 3 years ago
Xiaojing_liu wrote:
I tested in container:
liuxiaojing@linux-90dc:~> sudo podman run --rm -it registry.suse.de/home/okurz/container/images/openqa-review /bin/bash
According to gitlab log message:[32;1m$ sh -ex bin/openqa-review-daily-email $ARGS[0;m ++ echo '[32;1m$ sh -ex bin/openqa-review-daily-email $ARGS[0;m' ++ sh -ex bin/openqa-review-daily-email ++ dirname bin/openqa-review-daily-email + . bin/_common ++ set -o pipefail + '[' -z ']' + setup_tmpdir ++ mktemp -d + tmp=/tmp/tmp.uXWMAvvkIL + trap cleanup EXIT + openqa_host=https://openqa.suse.de + recv=openqa-suse-status@suse.de + html_target_file=openqa_suse_de_status.html + html_target=openqa_suse_de_status.html + openqa_review_args='--host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups ^(Released|Development|old)' + load_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL' + openqa_review_email_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL' + openqa_review_html_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL' + openqa_review_save_args='--reminder-comment-on-issues --save --save-dir /tmp/tmp.uXWMAvvkIL' ++ command -v openqa-review + openqa_review=/usr/bin/openqa-review ++ dirname bin/openqa-review-daily-email + TPL=bin/../dashboard_files/dashboard.html.in ++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.uXWMAvvkIL
It runs
/usr/bin/openqa-review ...
, this file belongs:32c2adb65251:/openqa_review # rpm -qf /usr/bin/openqa-review python38-openqa_review-1.20.2-1.3.noarch
this package doesn't include the pr https://github.com/os-autoinst/openqa_review/pull/139.
Not sure why we dogit clone https://github.com/os-autoinst/openqa_review.git .
in gitlab, we don't use these codes.
Should we update the container image to let it includes the change (https://github.com/os-autoinst/openqa_review/pull/139)? or change the gitlab-ci to use the github code?
We do clone the repo in GitLab from what I see here https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L37 - or did you mean something else?
But I wonder, why don't we make sure to update the image and use it everywhere?
Updated by Xiaojing_liu over 3 years ago
We do clone the repo in GitLab from what I see here https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L37 - or did you mean something else?
But I wonder, why don't we make sure to update the image and use it everywhere?
Yes, we clone the repo in GitLab. But I guess we don't use the github code to do openqa-review.
According to log message, it runs /usr/bin/openqa-review
, and we could see
32c2adb65251:/openqa_review # rpm -qf /usr/bin/openqa-review
python38-openqa_review-1.20.2-1.3.noarch
32c2adb65251:/openqa_review # rpm -ql python38-openqa_review-1.20.2-1.3.noarch
...
/usr/bin/openqa-review
...
/usr/lib/python3.8/site-packages/openqa_review/browser.py
/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py
So the github code isn't run, right? If I understand correctly, I wonder, why we clone github code here, we don't use it.
The latest container image is created on 29, Jan 2021 : https://registry.suse.de/cgi-bin/cooverview?srch_term=project%3D%5Ehome%3Aokurz#layers-17711ab9a6ab
Updated by livdywan over 3 years ago
Xiaojing_liu wrote:
We do clone the repo in GitLab from what I see here https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L37 - or did you mean something else?
But I wonder, why don't we make sure to update the image and use it everywhere?
Yes, we clone the repo in GitLab. But I guess we don't use the github code to do openqa-review.
According to log message, it runs/usr/bin/openqa-review
, and we could see32c2adb65251:/openqa_review # rpm -qf /usr/bin/openqa-review python38-openqa_review-1.20.2-1.3.noarch 32c2adb65251:/openqa_review # rpm -ql python38-openqa_review-1.20.2-1.3.noarch ... /usr/bin/openqa-review ... /usr/lib/python3.8/site-packages/openqa_review/browser.py /usr/lib/python3.8/site-packages/openqa_review/openqa_review.py
So the github code isn't run, right? If I understand correctly, I wonder, why we clone github code here, we don't use it.
Upon closer inspection I see that bin/openqa-review-daily-email
uses the repo, but said shellscript calls command -v openqa-review
unless $openqa_review
overrides it.
I'd vote to fix that default to call ./openqa_review/openqa_review.py
. You might be wrapping up your day by now... so I can offer proposing a branch to fix that.
Updated by livdywan over 3 years ago
cdywan wrote:
Upon closer inspection I see that
bin/openqa-review-daily-email
uses the repo, but said shellscript callscommand -v openqa-review
unless$openqa_review
overrides it.I'd vote to fix that default to call
./openqa_review/openqa_review.py
. You might be wrapping up your day by now... so I can offer proposing a branch to fix that.
Updated by okurz over 3 years ago
The problem is that I forgot to create a new package release for python-openqa_review which I can do the next days. As alternative you might be able to create a submission in OBS as well but I think we need a tag in the git repo first as well.
Updated by okurz over 3 years ago
- Copied to action #89536: Use openqa-review consistently from only package or only git, not mixed added
Updated by okurz over 3 years ago
So what I did, in a checkout of openqa_review:
git fetch origin
git co -b fix/private_comments origin/master
git tag -f 1.20.4 && git push -f origin 1.20.4 && git push -f origin HEAD:master
then in a checkout of https://build.opensuse.org/package/show/devel:languages:python/python-openqa_review
osc up && ./_update.sh
which created
https://build.opensuse.org/request/show/876941
I suggest we wait for that and see if a new image gets built. After that we check the new run of CI pipeline and see if the fix is effective. Another story what we identified is that it can appear inconsistent to use parts of scripts from a git checkout and parts from an installed binary as well. As using the package allows better use of our own products, read "eat your own dogfood", we think it's a good idea to keep using the package but then streamline the pipeline to not use a git checkout but rely on the package completely or switch to git-only -> #89536
Updated by Xiaojing_liu over 3 years ago
- Status changed from In Progress to Feedback
we are waiting for the new image result, it hasn't built successfully yet. So I change the status to feedback.
Updated by livdywan over 3 years ago
- Due date changed from 2021-03-10 to 2021-03-12
Updated by okurz over 3 years ago
- Status changed from Feedback to In Progress
the container image in https://build.suse.de/package/show/home:okurz:container/openqa-review failed to rebuild for some days. Apparently the problem is that OBS is unable to resolve file dependencies hence mutt as a dependency could not be installed. Fixed that within https://build.suse.de/projects/home:okurz:container/meta with
# Workaround for OBS being unable to resolve file dependencies, see discussion in https://matrix.to/#/!ZzdoIxkFyLxnxddLqM:matrix.org/$1615457617489563BXWEC:matrix.org?via=matrix.org
FileProvides: /usr/bin/zcat busybox-gzip
Updated by Xiaojing_liu over 3 years ago
- Status changed from In Progress to Resolved
- Estimated time set to 30.00 h
Retried the openqa_review pipeline after the container image was updated successfully by @okurz, there isn't error message 500 Server Error: Internal Server Error for url
in log file:
https://gitlab.suse.de/openqa/openqa-review/-/jobs/358710/raw
We also can see that the comments were added into Bugzilla: https://bugzilla.suse.com/buglist.cgi?quicksearch=commenter%3Aopenqa-review
So I set this ticket Resolved
Updated by Xiaojing_liu over 3 years ago
- Estimated time changed from 30.00 h to 40.00 h
Updated by Xiaojing_liu over 3 years ago
- Estimated time changed from 40.00 h to 60.00 h