Project

General

Profile

action #75214

openqa-review fails to post reminder comments on bugzilla, errors in log "Encountered error trying to post a reminder comment on issue"

Added by okurz 9 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-10-23
Due date:
2021-03-12
% Done:

0%

Estimated time:
60.00 h
Difficulty:

Description

Observation

from https://gitlab.suse.de/openqa/openqa-review/-/jobs/275810/raw

++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.pZbTFYl7W0
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ERROR:openqa_review.openqa_review:Encountered error trying to post a reminder comment on issue '* [jeos-base+phub_hyperv@svirt-hyperv-uefi](https://openqa.suse.de/tests/4846363 "Failed modules: firstrun"), [jeos-filesystem_hyperv@svirt-hyperv-uefi](https://openqa.suse.de/tests/4846366 "Failed modules: bootloader_uefi") -> [bsc#1176134](https://bugzilla.suse.com/show_bug.cgi?id=1176134 "[Build 20.3] JeOS images 15sp3 do not boot in uefi mode") (Ticket status: <span style="color: red;">VERIFIED (FIXED)</span>, prio/severity: P2/Normal, assignee: ms@suse.com)
': 500 Server Error: Internal Server Error for url: https://apibugzilla.suse.com/jsonrpc.cgi. Skipping.

Steps to reproduce

Be aware that the gitlab CI pipeline uses a dedicated user account as visible in https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L9 and https://gitlab.suse.de/openqa/openqa-review/-/settings/ci_cd (see "Variables")

Problem

Posting reminder comments from account "okurz" works. This is what we use so far on lord.arch.suse.de. Also it worked to post a reminder comment manually in https://bugzilla.suse.com/show_bug.cgi?id=1005824#c10 but so far posting from gitlab CI did not work

Suggestions

Further details

Initially reported in #52655#note-10


Related issues

Related to QA - action #52655: [epic] Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solutionBlocked2021-04-19

Copied to openQA Project - action #89536: Use openqa-review consistently from only package or only git, not mixedNew

History

#1 Updated by okurz 9 months ago

  • Related to action #52655: [epic] Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution added

#2 Updated by Xiaojing_liu 5 months ago

  • Status changed from Workable to In Progress
  • Assignee set to Xiaojing_liu

I would like to try this. Also it's a chance to learn Python.

#3 Updated by openqa_review 5 months ago

  • Due date set to 2021-03-10

Setting due date based on mean cycle time of SUSE QE Tools

#4 Updated by Xiaojing_liu 5 months ago

I did a test using openqa_review to get comments from https://bugzilla.suse.com/show_bug.cgi?id=1005824. it only can get the public comments. I tried this use my own account, all comments I could get. So I think openqa_review does not have the permission. After I changed the is_private to False, the comment was added successfully. See: https://bugzilla.suse.com/show_bug.cgi?id=1005824 comments 11 and 13.
Not sure if we should upgrade openqa_review permission, I created a pr to change the parameter: https://github.com/os-autoinst/openqa_review/pull/139. If we need to update the permission, I can close the pull request.

#5 Updated by Xiaojing_liu 5 months ago

I run openqa-review locally
python2.7 -m openqa_review.openqa_review --host http://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --job-groups '15 SP3' --reminder-comment-on-issues

After changing the is_private : False, the comment was added to bugzilla successfully.
see: https://bugzilla.suse.com/show_bug.cgi?id=1179434 comment#27

But CI pipeline still report error 500: 500 Server Error: Internal Server Error for url : https://gitlab.suse.de/openqa/openqa-review/-/jobs/349401/raw

#6 Updated by Xiaojing_liu 5 months ago

I tested in container:
liuxiaojing@linux-90dc:~> sudo podman run --rm -it registry.suse.de/home/okurz/container/images/openqa-review /bin/bash
According to gitlab log message:

$ sh -ex bin/openqa-review-daily-email $ARGS
++ echo '$ sh -ex bin/openqa-review-daily-email $ARGS'
++ sh -ex bin/openqa-review-daily-email
++ dirname bin/openqa-review-daily-email
+ . bin/_common
++ set -o pipefail
+ '[' -z ']'
+ setup_tmpdir
++ mktemp -d
+ tmp=/tmp/tmp.uXWMAvvkIL
+ trap cleanup EXIT
+ openqa_host=https://openqa.suse.de
+ recv=openqa-suse-status@suse.de
+ html_target_file=openqa_suse_de_status.html
+ html_target=openqa_suse_de_status.html
+ openqa_review_args='--host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups ^(Released|Development|old)'
+ load_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_email_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_html_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_save_args='--reminder-comment-on-issues --save --save-dir /tmp/tmp.uXWMAvvkIL'
++ command -v openqa-review
+ openqa_review=/usr/bin/openqa-review
++ dirname bin/openqa-review-daily-email
+ TPL=bin/../dashboard_files/dashboard.html.in
++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.uXWMAvvkIL

It runs /usr/bin/openqa-review ..., this file belongs:

32c2adb65251:/openqa_review # rpm -qf /usr/bin/openqa-review
python38-openqa_review-1.20.2-1.3.noarch

this package doesn't include the pr https://github.com/os-autoinst/openqa_review/pull/139.
Not sure why we do git clone https://github.com/os-autoinst/openqa_review.git . in gitlab, we don't use these codes.
Should we update the container image to let it includes the change (https://github.com/os-autoinst/openqa_review/pull/139)? or change the gitlab-ci to use the github code?

#7 Updated by cdywan 5 months ago

Xiaojing_liu wrote:

I tested in container:
liuxiaojing@linux-90dc:~> sudo podman run --rm -it registry.suse.de/home/okurz/container/images/openqa-review /bin/bash
According to gitlab log message:

$ sh -ex bin/openqa-review-daily-email $ARGS
++ echo '$ sh -ex bin/openqa-review-daily-email $ARGS'
++ sh -ex bin/openqa-review-daily-email
++ dirname bin/openqa-review-daily-email
+ . bin/_common
++ set -o pipefail
+ '[' -z ']'
+ setup_tmpdir
++ mktemp -d
+ tmp=/tmp/tmp.uXWMAvvkIL
+ trap cleanup EXIT
+ openqa_host=https://openqa.suse.de
+ recv=openqa-suse-status@suse.de
+ html_target_file=openqa_suse_de_status.html
+ html_target=openqa_suse_de_status.html
+ openqa_review_args='--host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups ^(Released|Development|old)'
+ load_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_email_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_html_args='--load --load-dir=/tmp/tmp.uXWMAvvkIL'
+ openqa_review_save_args='--reminder-comment-on-issues --save --save-dir /tmp/tmp.uXWMAvvkIL'
++ command -v openqa-review
+ openqa_review=/usr/bin/openqa-review
++ dirname bin/openqa-review-daily-email
+ TPL=bin/../dashboard_files/dashboard.html.in
++ /usr/bin/openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old)' --reminder-comment-on-issues --save --save-dir /tmp/tmp.uXWMAvvkIL

It runs /usr/bin/openqa-review ..., this file belongs:

32c2adb65251:/openqa_review # rpm -qf /usr/bin/openqa-review
python38-openqa_review-1.20.2-1.3.noarch

this package doesn't include the pr https://github.com/os-autoinst/openqa_review/pull/139.
Not sure why we do git clone https://github.com/os-autoinst/openqa_review.git . in gitlab, we don't use these codes.
Should we update the container image to let it includes the change (https://github.com/os-autoinst/openqa_review/pull/139)? or change the gitlab-ci to use the github code?

We do clone the repo in GitLab from what I see here https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L37 - or did you mean something else?

But I wonder, why don't we make sure to update the image and use it everywhere?

#8 Updated by Xiaojing_liu 5 months ago

We do clone the repo in GitLab from what I see here https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L37 - or did you mean something else?

But I wonder, why don't we make sure to update the image and use it everywhere?

Yes, we clone the repo in GitLab. But I guess we don't use the github code to do openqa-review.
According to log message, it runs /usr/bin/openqa-review, and we could see

32c2adb65251:/openqa_review # rpm -qf /usr/bin/openqa-review
python38-openqa_review-1.20.2-1.3.noarch
32c2adb65251:/openqa_review # rpm -ql python38-openqa_review-1.20.2-1.3.noarch
...
/usr/bin/openqa-review
...
/usr/lib/python3.8/site-packages/openqa_review/browser.py
/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py

So the github code isn't run, right? If I understand correctly, I wonder, why we clone github code here, we don't use it.

The latest container image is created on 29, Jan 2021 : https://registry.suse.de/cgi-bin/cooverview?srch_term=project%3D%5Ehome%3Aokurz#layers-17711ab9a6ab

#9 Updated by cdywan 5 months ago

Xiaojing_liu wrote:

We do clone the repo in GitLab from what I see here https://gitlab.suse.de/openqa/openqa-review/-/blob/master/.gitlab-ci.yml#L37 - or did you mean something else?

But I wonder, why don't we make sure to update the image and use it everywhere?

Yes, we clone the repo in GitLab. But I guess we don't use the github code to do openqa-review.
According to log message, it runs /usr/bin/openqa-review, and we could see

32c2adb65251:/openqa_review # rpm -qf /usr/bin/openqa-review
python38-openqa_review-1.20.2-1.3.noarch
32c2adb65251:/openqa_review # rpm -ql python38-openqa_review-1.20.2-1.3.noarch
...
/usr/bin/openqa-review
...
/usr/lib/python3.8/site-packages/openqa_review/browser.py
/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py

So the github code isn't run, right? If I understand correctly, I wonder, why we clone github code here, we don't use it.

Upon closer inspection I see that bin/openqa-review-daily-email uses the repo, but said shellscript calls command -v openqa-review unless $openqa_review overrides it.

I'd vote to fix that default to call ./openqa_review/openqa_review.py. You might be wrapping up your day by now... so I can offer proposing a branch to fix that.

#10 Updated by cdywan 5 months ago

cdywan wrote:

Upon closer inspection I see that bin/openqa-review-daily-email uses the repo, but said shellscript calls command -v openqa-review unless $openqa_review overrides it.

I'd vote to fix that default to call ./openqa_review/openqa_review.py. You might be wrapping up your day by now... so I can offer proposing a branch to fix that.

https://github.com/os-autoinst/openqa_review/pull/141

#11 Updated by okurz 5 months ago

The problem is that I forgot to create a new package release for python-openqa_review which I can do the next days. As alternative you might be able to create a submission in OBS as well but I think we need a tag in the git repo first as well.

#12 Updated by okurz 5 months ago

  • Copied to action #89536: Use openqa-review consistently from only package or only git, not mixed added

#13 Updated by okurz 5 months ago

So what I did, in a checkout of openqa_review:

git fetch origin
git co -b fix/private_comments origin/master
git tag -f 1.20.4 && git push -f origin 1.20.4 && git push -f origin HEAD:master

then in a checkout of https://build.opensuse.org/package/show/devel:languages:python/python-openqa_review

osc up && ./_update.sh

which created
https://build.opensuse.org/request/show/876941
I suggest we wait for that and see if a new image gets built. After that we check the new run of CI pipeline and see if the fix is effective. Another story what we identified is that it can appear inconsistent to use parts of scripts from a git checkout and parts from an installed binary as well. As using the package allows better use of our own products, read "eat your own dogfood", we think it's a good idea to keep using the package but then streamline the pipeline to not use a git checkout but rely on the package completely or switch to git-only -> #89536

#14 Updated by Xiaojing_liu 5 months ago

  • Status changed from In Progress to Feedback

we are waiting for the new image result, it hasn't built successfully yet. So I change the status to feedback.

#15 Updated by cdywan 5 months ago

  • Due date changed from 2021-03-10 to 2021-03-12

#16 Updated by okurz 5 months ago

  • Status changed from Feedback to In Progress

the container image in https://build.suse.de/package/show/home:okurz:container/openqa-review failed to rebuild for some days. Apparently the problem is that OBS is unable to resolve file dependencies hence mutt as a dependency could not be installed. Fixed that within https://build.suse.de/projects/home:okurz:container/meta with

# Workaround for OBS being unable to resolve file dependencies, see discussion in https://matrix.to/#/!ZzdoIxkFyLxnxddLqM:matrix.org/$1615457617489563BXWEC:matrix.org?via=matrix.org
FileProvides: /usr/bin/zcat busybox-gzip

#17 Updated by Xiaojing_liu 5 months ago

  • Status changed from In Progress to Resolved
  • Estimated time set to 30.00 h

Retried the openqa_review pipeline after the container image was updated successfully by okurz, there isn't error message 500 Server Error: Internal Server Error for url in log file:
https://gitlab.suse.de/openqa/openqa-review/-/jobs/358710/raw
We also can see that the comments were added into Bugzilla: https://bugzilla.suse.com/buglist.cgi?quicksearch=commenter%3Aopenqa-review
So I set this ticket Resolved

#18 Updated by Xiaojing_liu 5 months ago

  • Estimated time changed from 30.00 h to 40.00 h

#19 Updated by okurz 5 months ago

Thanks. Awesome achievement!

#20 Updated by Xiaojing_liu 3 months ago

  • Estimated time changed from 40.00 h to 60.00 h

Also available in: Atom PDF