Project

General

Profile

Actions

action #52655

closed

openQA Project (public) - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

[epic] Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution

Added by okurz over 5 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Start date:
2021-04-19
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Motivation

Currently the reports are generated on lord.arch by the user "okurz". We can look for a more sustainable long-term solution

Suggestions

we could run openqa-review

  • in gitlab CI
  • on GOCD at botmaster.suse.de
  • on a QA SLE VM

for "gocd", from #openqa-dev (rc):

I asked coolo "WDYT running "openqa-review" which generates e.g. https://w3.nue.suse.com/~okurz/openqa_sle15_status.html on botmaster.suse.de GOCD? Would you accept a PR adding that to GOCD? In this case, what else would I need to test it out and get it to run?".
He is fine to run it on botmaster.suse.de. I am checking the deps with the container def. https://build.opensuse.org/package/view_file/openSUSE:Tools:Images/gocd-agent-release-tools/Dockerfile?expand=1 is installed. `zypper -n in python3-openqa_review && openqa-review …" would not work. The scripts run as user "go". Either the script is part of the container (and won't change) - or it should run from a git. The containers are running constantly - they are not per job. The container change on deployment of gocd, they only contain dependencies and the gocd agent. If we need new dependencies, I can inject zypper -n calls on every container so run from git checkout - this way you also track which job ran on which revision. Basically all of them run from git checkout, https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/autosubmit.gocd.yaml is a good example. I would need additional deps "python3-appdirs python3-beautifulsoup4 python3-future python3-humanfriendly python3-setuptools python3-sortedcontainers python3-soupsieve" to make it run directly from git.
I am not fully convinced to go to botmaster.suse.de with gocd considering that this would mean to not use the RPM package but run from git but build the container with missing dependencies which I am not sure if it's the right approach.


Subtasks 2 (0 open2 closed)

action #91356: Save openqa-review reports as gitlab CI artifactsResolvedosukup2021-04-19

Actions
action #93710: Reference individual openqa-review reports in gitlab CI artifacts, e.g. using gitlab pagesResolvedlivdywan

Actions

Related issues 2 (0 open2 closed)

Related to openQA Project (public) - action #75214: openqa-review fails to post reminder comments on bugzilla, errors in log "Encountered error trying to post a reminder comment on issue"ResolvedXiaojing_liu2020-10-232021-03-12

Actions
Related to openQA Project (public) - coordination #91914: [epic] Make reviewing openQA results per squad easierResolvedokurz2021-05-25

Actions
Actions #2

Updated by okurz about 5 years ago

  • Status changed from New to In Progress

By now I would consider building a custom container and using that in gitlab CI a viable alternative, e.g. as in https://gitlab.suse.de/openqa/auto-review/

Building container in https://build.suse.de/package/show/home:okurz:container/openqa-review with all dependencies and trying out in https://gitlab.suse.de/okurz/openqa-review . Created https://gitlab.suse.de/openqa/openqa-review already.

I would need to put variables like a redmine API key and bugzilla username and password into gitlab config and I would like to avoid my personal user password. hmuelle gave me a bugzilla bot account "QASLE" but first the name is too specific and the email is qa-sle-mgmt. I doubt that ML is a good target. I should ask if we can have a better account created.

EDIT: 2019-11-19: Asked in https://chat.suse.de/channel/suse-it-ama?msg=pXMoXFKaEnk5AfWGw "hi, I would like to have a non-personal bugzilla account backed by a mailing list so that a script from gitlab CI pipeline can post comments which previously I had done with my personal user account but I do not want to save my personal password in gitlab CI variables. Any idea who can help with that?"

Actions #3

Updated by okurz about 5 years ago

  • Due date set to 2019-11-26
  • Status changed from In Progress to Feedback

Setting to Feedback with reminder due-date to see if I got any response on question in chat.

EDIT: 2019-11-19: fcrozat was kind enough to answer directly, pointed me to RIO ticket. I created a RIO ticket REQ_333370 with content:

hi, I would like to have a non-personal bugzilla account backed by a mailing list so that a script from gitlab CI pipeline can post comments which previously I had done with my personal user account but I do not want to save my personal password in gitlab CI variables.

Preferences:

username: openqa-review
mailing list: openqa-review@suse.de

I already created a ticket for SUSE Engineering Infrastructure to create the mailing list: https://infra.nue.suse.com/SelfService/Display.html?id=154691

EDIT: 2019-11-19: The mailing list was already created: openqa-review@suse.de

Actions #4

Updated by okurz about 5 years ago

  • Due date changed from 2019-11-26 to 2019-12-17

no answer in RIO ticket, giving more time.

Actions #5

Updated by okurz about 5 years ago

  • Status changed from Feedback to Blocked
Actions #6

Updated by okurz about 5 years ago

  • Due date changed from 2019-12-17 to 2020-01-17

No response :,( . Asked in https://chat.suse.de/channel/suse-it-ama?msg=7AxwXTow2BijJqLo8 if I can do anything better and added a note in the RIO request.

Actions #7

Updated by okurz almost 5 years ago

  • Due date changed from 2020-01-17 to 2020-02-19

Pinged again in RC with https://chat.suse.de/channel/suse-it-ama?msg=diXTQaNo35oPWuo49 : "Zuzana van Wijk Vodova hi, you offered your help for REQ_333370 last month and asked me to ping you directly for escalation. Unfortunately there has not been any movement with the request since then. Is there anything I can do myself to move this forward?"

Actions #8

Updated by okurz almost 5 years ago

  • Status changed from Blocked to Feedback

(Finally) received an update in https://rio.microfocuscloud.com/tmtrack/tmtrack.dll?shell=swc&IssuePage&Template=viewwrapper&RecordId=113050&TableId=1003#1 .

C   Chris Slade  -  23/01/2020 10:55:03 PM   -(public)  Send E-mail      
    Note from Chris Slade about REQ_333370 [ttid: 1003,113050]

 Message Address Details
Message Addressed To: Oliver Kurz

Oliver,



There is no such thing as a Bugzilla account. Bugzilla uses the external customer accounts that are created with our self reg service. So you would need to create an external account using that email address. to create a mailing list you will need to contact someone in charge of mailing lists. I believe that Karol Babioch is in charge of the suse.de mailing lists.



Thank you,

Chris Slade

So I created an account "openqa_review" over https://login.microfocus.com , password stored internally in my password manager. I could validate the email address of the mailing list openqa-review@suse.de and proceed now. Over https://bugzilla.suse.com/index.cgi?GoAheadAndLogIn=1 I could not yet validate the account so far unfortunately. Maybe I need to give it some time. Can retry another day.

EDIT: Initialized an account on progress.o.o which also created an API key. I added the API key as well as username and password for bugzilla into https://gitlab.suse.de/openqa/openqa-review/-/settings/ci_cd and started https://gitlab.suse.de/openqa/openqa-review/-/jobs/163830 . I can check for reminder comments on bugzilla with http://fastzilla.suse.de/?q=commenter%3Aopenqa-review%40suse.de . Job was aborted after timeout of 1h, bumped to 4h in gitlab CI config for this project.

https://gitlab.suse.de/openqa/openqa-review/-/jobs/163908 succceeded after 118m.

I added some pipeline schedules on
https://gitlab.suse.de/openqa/openqa-review/pipeline_schedules but after temporary network problems within the network I deactivated the schedules again for now.

Open point: I need to publish the generated report. Previously we just saved to /suse/okurz which isn't available within gitlab CI. We could scp the file out but I prefer to save the report as gitlab CI artifacts assuming I can reference the reports accordingly by URL as documented on https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html and at best update the short links to the reports within s.qa's db.

Actions #9

Updated by okurz over 4 years ago

  • Due date deleted (2020-02-19)
  • Status changed from Feedback to Workable

I can try accessing job artifacts with static URLs anytime.

Actions #10

Updated by okurz over 4 years ago

gitlab CI shows errors as reported in #66868 and visible in https://gitlab.suse.de/openqa/openqa-review/-/jobs , e.g. raw logs show "ERROR:openqa_review.openqa_review:Encountered error trying to post a reminder comment on issue '* soft fails: ltp_dio, ltp_openposix, ltp_sched, ltp_syscalls_ipc -> bsc#1158145: Machine didn't shut down within 60 sec (Ticket status: NEW, prio/severity: P5/Normal, assignee: screening-team-bugs@suse.de) ': 500 Server Error: Internal Server Error for url: https://apibugzilla.suse.com/jsonrpc.cgi. Skipping."

maybe this never worked so far for openqa-review?

With the bugzilla instance being moved to SUSE Nbg the performance is way better for searching so we can look on https://bugzilla.suse.com/buglist.cgi?quicksearch=commenter%3Aopenqa-review

Idea as next step: Try to read/post bugs over API as openqa-review, e.g. in test bug https://bugzilla.suse.com/show_bug.cgi?id=1005824

Actions #11

Updated by okurz about 4 years ago

  • Related to action #75214: openqa-review fails to post reminder comments on bugzilla, errors in log "Encountered error trying to post a reminder comment on issue" added
Actions #12

Updated by okurz about 4 years ago

  • Status changed from Workable to Blocked
  • Target version set to Ready

waiting for #75214 first

Actions #13

Updated by okurz about 4 years ago

  • Estimated time set to 80142.00 h
Actions #14

Updated by okurz about 4 years ago

  • Estimated time deleted (80142.00 h)
Actions #15

Updated by okurz almost 4 years ago

  • Status changed from Blocked to Workable

With #75214 resolved I can look again what is left to be moved from lord.arch to gitlab CI jobs

Actions #16

Updated by okurz over 3 years ago

it seems that the scheduled CI jobs within gitlab commonly run a bit later than the cron jobs on lord so I will turn that around so that the lord.arch cron jobs are only mostly "fallback" to have only a tiny delta between the times when the jobs trigger.

Actions #17

Updated by okurz over 3 years ago

  • Subject changed from Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution to [epic] Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution
  • Status changed from Workable to Blocked
Actions #18

Updated by okurz over 3 years ago

  • Parent task set to #39719
Actions #19

Updated by livdywan over 3 years ago

  • Description updated (diff)
Actions #20

Updated by okurz over 3 years ago

Actions #21

Updated by okurz over 3 years ago

  • Project changed from 46 to QA (public)

Moving to "QA" project as I would like to simplify access to others even though this is a "SUSE-internal task".

Actions #22

Updated by okurz over 3 years ago

  • Status changed from Blocked to Resolved

All openqa-review cron jobs have been replaced by gitlab CI schedules in https://gitlab.suse.de/openqa/openqa-review/-/pipeline_schedules with the results visible in https://openqa.io.suse.de/openqa-review/

Actions

Also available in: Atom PDF