Project

General

Profile

action #52655

openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

[epic] Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution

Added by okurz over 2 years ago. Updated 22 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2021-04-19
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

Motivation

Currently the reports are generated on lord.arch by the user "okurz". We can look for a more sustainable long-term solution

Suggestions

we could run openqa-review

  • in gitlab CI
  • on GOCD at botmaster.suse.de
  • on a QA SLE VM

for "gocd", from #openqa-dev (rc):

I asked coolo "WDYT running "openqa-review" which generates e.g. https://w3.nue.suse.com/~okurz/openqa_sle15_status.html on botmaster.suse.de GOCD? Would you accept a PR adding that to GOCD? In this case, what else would I need to test it out and get it to run?".
He is fine to run it on botmaster.suse.de. I am checking the deps with the container def. https://build.opensuse.org/package/view_file/openSUSE:Tools:Images/gocd-agent-release-tools/Dockerfile?expand=1 is installed. `zypper -n in python3-openqa_review && openqa-review …" would not work. The scripts run as user "go". Either the script is part of the container (and won't change) - or it should run from a git. The containers are running constantly - they are not per job. The container change on deployment of gocd, they only contain dependencies and the gocd agent. If we need new dependencies, I can inject zypper -n calls on every container so run from git checkout - this way you also track which job ran on which revision. Basically all of them run from git checkout, https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/autosubmit.gocd.yaml is a good example. I would need additional deps "python3-appdirs python3-beautifulsoup4 python3-future python3-humanfriendly python3-setuptools python3-sortedcontainers python3-soupsieve" to make it run directly from git.
I am not fully convinced to go to botmaster.suse.de with gocd considering that this would mean to not use the RPM package but run from git but build the container with missing dependencies which I am not sure if it's the right approach.


Subtasks

action #91356: Save openqa-review reports as gitlab CI artifactsResolvedosukup

action #93710: Reference individual openqa-review reports in gitlab CI artifacts, e.g. using gitlab pagesResolvedcdywan


Related issues

Related to openQA Project - action #75214: openqa-review fails to post reminder comments on bugzilla, errors in log "Encountered error trying to post a reminder comment on issue"Resolved2020-10-232021-03-12

Related to openQA Project - coordination #91914: [epic] Make reviewing openQA results per squad easierBlocked2016-02-09

History

#2 Updated by okurz almost 2 years ago

  • Status changed from New to In Progress

By now I would consider building a custom container and using that in gitlab CI a viable alternative, e.g. as in https://gitlab.suse.de/openqa/auto-review/

Building container in https://build.suse.de/package/show/home:okurz:container/openqa-review with all dependencies and trying out in https://gitlab.suse.de/okurz/openqa-review . Created https://gitlab.suse.de/openqa/openqa-review already.

I would need to put variables like a redmine API key and bugzilla username and password into gitlab config and I would like to avoid my personal user password. hmuelle gave me a bugzilla bot account "QASLE" but first the name is too specific and the email is qa-sle-mgmt. I doubt that ML is a good target. I should ask if we can have a better account created.

EDIT: 2019-11-19: Asked in https://chat.suse.de/channel/suse-it-ama?msg=pXMoXFKaEnk5AfWGw "hi, I would like to have a non-personal bugzilla account backed by a mailing list so that a script from gitlab CI pipeline can post comments which previously I had done with my personal user account but I do not want to save my personal password in gitlab CI variables. Any idea who can help with that?"

#3 Updated by okurz almost 2 years ago

  • Due date set to 2019-11-26
  • Status changed from In Progress to Feedback

Setting to Feedback with reminder due-date to see if I got any response on question in chat.

EDIT: 2019-11-19: fcrozat was kind enough to answer directly, pointed me to RIO ticket. I created a RIO ticket REQ_333370 with content:

hi, I would like to have a non-personal bugzilla account backed by a mailing list so that a script from gitlab CI pipeline can post comments which previously I had done with my personal user account but I do not want to save my personal password in gitlab CI variables.

Preferences:

username: openqa-review
mailing list: openqa-review@suse.de

I already created a ticket for SUSE Engineering Infrastructure to create the mailing list: https://infra.nue.suse.com/SelfService/Display.html?id=154691

EDIT: 2019-11-19: The mailing list was already created: openqa-review@suse.de

#4 Updated by okurz almost 2 years ago

  • Due date changed from 2019-11-26 to 2019-12-17

no answer in RIO ticket, giving more time.

#5 Updated by okurz almost 2 years ago

  • Status changed from Feedback to Blocked

#6 Updated by okurz almost 2 years ago

  • Due date changed from 2019-12-17 to 2020-01-17

No response :,( . Asked in https://chat.suse.de/channel/suse-it-ama?msg=7AxwXTow2BijJqLo8 if I can do anything better and added a note in the RIO request.

#7 Updated by okurz over 1 year ago

  • Due date changed from 2020-01-17 to 2020-02-19

Pinged again in RC with https://chat.suse.de/channel/suse-it-ama?msg=diXTQaNo35oPWuo49 : "Zuzana van Wijk Vodova hi, you offered your help for REQ_333370 last month and asked me to ping you directly for escalation. Unfortunately there has not been any movement with the request since then. Is there anything I can do myself to move this forward?"

#8 Updated by okurz over 1 year ago

  • Status changed from Blocked to Feedback

(Finally) received an update in https://rio.microfocuscloud.com/tmtrack/tmtrack.dll?shell=swc&IssuePage&Template=viewwrapper&RecordId=113050&TableId=1003#1 .

C   Chris Slade  -  23/01/2020 10:55:03 PM   -(public)  Send E-mail      
    Note from Chris Slade about REQ_333370 [ttid: 1003,113050]

 Message Address Details
Message Addressed To: Oliver Kurz

Oliver,



There is no such thing as a Bugzilla account. Bugzilla uses the external customer accounts that are created with our self reg service. So you would need to create an external account using that email address. to create a mailing list you will need to contact someone in charge of mailing lists. I believe that Karol Babioch is in charge of the suse.de mailing lists.



Thank you,

Chris Slade

So I created an account "openqa_review" over https://login.microfocus.com , password stored internally in my password manager. I could validate the email address of the mailing list openqa-review@suse.de and proceed now. Over https://bugzilla.suse.com/index.cgi?GoAheadAndLogIn=1 I could not yet validate the account so far unfortunately. Maybe I need to give it some time. Can retry another day.

EDIT: Initialized an account on progress.o.o which also created an API key. I added the API key as well as username and password for bugzilla into https://gitlab.suse.de/openqa/openqa-review/-/settings/ci_cd and started https://gitlab.suse.de/openqa/openqa-review/-/jobs/163830 . I can check for reminder comments on bugzilla with http://fastzilla.suse.de/?q=commenter%3Aopenqa-review%40suse.de . Job was aborted after timeout of 1h, bumped to 4h in gitlab CI config for this project.

https://gitlab.suse.de/openqa/openqa-review/-/jobs/163908 succceeded after 118m.

I added some pipeline schedules on
https://gitlab.suse.de/openqa/openqa-review/pipeline_schedules but after temporary network problems within the network I deactivated the schedules again for now.

Open point: I need to publish the generated report. Previously we just saved to /suse/okurz which isn't available within gitlab CI. We could scp the file out but I prefer to save the report as gitlab CI artifacts assuming I can reference the reports accordingly by URL as documented on https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html and at best update the short links to the reports within s.qa's db.

#9 Updated by okurz over 1 year ago

  • Due date deleted (2020-02-19)
  • Status changed from Feedback to Workable

I can try accessing job artifacts with static URLs anytime.

#10 Updated by okurz over 1 year ago

gitlab CI shows errors as reported in #66868 and visible in https://gitlab.suse.de/openqa/openqa-review/-/jobs , e.g. raw logs show "ERROR:openqa_review.openqa_review:Encountered error trying to post a reminder comment on issue '* soft fails: ltp_dio, ltp_openposix, ltp_sched, ltp_syscalls_ipc -> bsc#1158145: Machine didn't shut down within 60 sec (Ticket status: NEW, prio/severity: P5/Normal, assignee: screening-team-bugs@suse.de) ': 500 Server Error: Internal Server Error for url: https://apibugzilla.suse.com/jsonrpc.cgi. Skipping."

maybe this never worked so far for openqa-review?

With the bugzilla instance being moved to SUSE Nbg the performance is way better for searching so we can look on https://bugzilla.suse.com/buglist.cgi?quicksearch=commenter%3Aopenqa-review

Idea as next step: Try to read/post bugs over API as openqa-review, e.g. in test bug https://bugzilla.suse.com/show_bug.cgi?id=1005824

#11 Updated by okurz 11 months ago

  • Related to action #75214: openqa-review fails to post reminder comments on bugzilla, errors in log "Encountered error trying to post a reminder comment on issue" added

#12 Updated by okurz 11 months ago

  • Status changed from Workable to Blocked
  • Target version set to Ready

waiting for #75214 first

#13 Updated by okurz 10 months ago

  • Estimated time set to 80142.00 h

#14 Updated by okurz 10 months ago

  • Estimated time deleted (80142.00 h)

#15 Updated by okurz 7 months ago

  • Status changed from Blocked to Workable

With #75214 resolved I can look again what is left to be moved from lord.arch to gitlab CI jobs

#16 Updated by okurz 5 months ago

it seems that the scheduled CI jobs within gitlab commonly run a bit later than the cron jobs on lord so I will turn that around so that the lord.arch cron jobs are only mostly "fallback" to have only a tiny delta between the times when the jobs trigger.

#17 Updated by okurz 5 months ago

  • Subject changed from Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution to [epic] Move openqa-review from cron-jobs on lord.arch to a more sustainable long-term solution
  • Status changed from Workable to Blocked

#18 Updated by okurz 5 months ago

  • Parent task set to #39719

#19 Updated by cdywan 4 months ago

  • Description updated (diff)

#20 Updated by okurz 3 months ago

#21 Updated by okurz 3 months ago

  • Project changed from SUSE QA to QA

Moving to "QA" project as I would like to simplify access to others even though this is a "SUSE-internal task".

#22 Updated by okurz 22 days ago

  • Status changed from Blocked to Resolved

All openqa-review cron jobs have been replaced by gitlab CI schedules in https://gitlab.suse.de/openqa/openqa-review/-/pipeline_schedules with the results visible in https://openqa.io.suse.de/openqa-review/

Also available in: Atom PDF