action #47441

[functional][u] test code feedback - integrate bots to test distribution on github

Added by szarate about 1 year ago. Updated about 1 month ago.

Status:WorkableStart date:12/02/2019
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-Estimated time:42.00 hours
Target version:SUSE QA tests - Milestone 31
Duration:

Description

Currently there's the problem with Pull requests sometimes being forgotten and open forever, leading to tickets that are open for too long just because a PR wasn't merged in time.

There's the proposal of using a bot that upon certain syntax either pings the person on irc, or better writes on an irc channel (#testing or #opensuse-factory? or #opensuse-someotherchannel? without being too noisy...

There are many bots that are available for this:

Requirements for an acceptable bot:

The bot shall be able to fulfill the Acceptance Criteria post timebox.

These are suggestions, but definitely not limited to those. A quick test should allow us to move forward and pick what's best for the team.

Suggestions

  • Give a quick check to the options given or look for some alternatives
  • Test the bot, ideally it will react when a PR has more than X days without progress, and when it is mentioned or when it reads a comment with certain syntax.

Tasks:

Acceptance Criteria: Post timebox

  • AC.1: Bot is running and setup is documented in a wiki
  • AC.2: PR older than 60 days without activity are considered stale
  • AC.3: PR that is already marked as stale, with over 7 days of inactivity are closed
  • AC.4: PR with labels RFC are excempt (can be expanded later)
  • AC.5: Stale PR are marked with STALE label

Suggestions

To make it easier to test, it can be initially set up in the person's own github fork of os-autoinst-distri-opensuse (or any other repo) with very short times


Related issues

Related to openQA Tests - action #48392: [functional][y][spike/research] self-tests in os-autoinst... Resolved 25/02/2019 12/03/2019

History

#1 Updated by okurz about 1 year ago

  • Target version changed from Milestone 22 to Milestone 24

Actually M22 is over. Let's focus a bit more on the existing tickets, when we trimmed down the backlog a bit we can pick this ok, shall we?

I don't understand how you see an 8h timebox + the ACs feasible. Did you confuse timebox and estimate? For a timebox I would expect as single AC that the findings are documented.

#2 Updated by szarate about 1 year ago

  • Description updated (diff)

I don't understand how you see an 8h timebox + the ACs feasible. Did you confuse timebox and estimate? For a timebox I would expect as single AC that the findings are documented.

We agreed to this during the retrospective. I don't expect all of the AC's to be closed within one iteration, let's update the description :)

#3 Updated by okurz about 1 year ago

  • Target version changed from Milestone 24 to Milestone 23

I see, that makes perfect sense. Thank you. So let's try in M23 because it is asked for by the team :)

#4 Updated by okurz about 1 year ago

  • Due date set to 26/03/2019

#5 Updated by szarate about 1 year ago

  • Status changed from New to Workable

#6 Updated by okurz 12 months ago

  • Related to action #48392: [functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule added

#7 Updated by jorauch 12 months ago

  • Assignee set to jorauch

That looks lovely

#8 Updated by okurz 12 months ago

  • Subject changed from [functional][u][y][timebox:8h] test code feedback - integrate bots to test distribution on github to [functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github

As QSF-u has picked it up first, let them handle it :)

#9 Updated by jorauch 12 months ago

This one also looks useful:
https://github.com/probot/stale

#10 Updated by jorauch 11 months ago

  • Status changed from Workable to In Progress

actually longer in progress
I will write a new one in Python, as the examples do not perfectly fit our needs

#11 Updated by jorauch 11 months ago

The way to go is:

g = GitHub instance
org = g.get_organization('os-autoinst')
repo = org.get_repo('os-autoinst-distri-opensuse')
pr = repo.get_pull(numberOfPR)

#12 Updated by jorauch 11 months ago

Repo with the script:
https://github.com/DrMullings/Scripts-Snippets-Stuff/blob/master/Marvin/marvin.py

Progress so far:

  • able to age of all PRs
  • able to get age of comment of all PRs
  • able to get if a PR is approved

#13 Updated by jorauch 11 months ago

The bot is now able to connect to IRC and GitHub, it can collect the data of all PRs and detect PRs in IRC
Need to talk to szarate later about how this should work exactly since just printing the PR state to IRC would be quite a mess

#14 Updated by jorauch 11 months ago

As discussed with szarate:

  • Bot shall scan PRs and consider them stalled by a preconfigured parameter (e.g. 3-5 days for OpenQA Engineers, 2 weeks for external guys)
  • based on the List of old PRs we write a weekly comment, mentioning last commentator to GitHub (Bot Comments, and Author Comments do not count)
  • we can print a list of the old PRs to an IRC channel (best packaged in a command, e.g. "/bot give PRs older than 5 days")
  • Bot can print the age of mentioned PRs in a channel

#15 Updated by jorauch 11 months ago

Stuff can now be found here:
https://github.com/DrMullings/Scripts-Snippets-Stuff/tree/master/Marvin

  • It is not yet capable of differentiating between internal and external guys (we should re-discuss this anyway, it's hard to do from a technical side)
  • It can write comments on a PR after the staletime is over
  • The list of PRs could be printed to the IRC, this is not yet implemented though
  • It checks the age (of the last comment) of posted PRs

I still need write access ID and Secret for os-autoinst to get it productive

#16 Updated by SLindoMansilla 11 months ago

As spoken with jrauch,

I think that a bot writing comments in the PRs will cause a lot of noise. People who didn't care about reviewing them will ignore those emails, or even complain (like it happened with bugzilla tickets).

Writing reminder in an IRC chat is better, since only people in that chat will receive the messages. We need to define yet who is taking care of reviewing PR. I think that our old model "each one does when she/he has time" has been proven to not work. We need a dedicated group of people for that. And, if at the end, this person will be only me, the bot will cause only noise, since I will ignore it and continue with my approach:

  1. Look for tickets from U-Team on feedback
  2. If it has a PR, review (eventually merging it)

We need to think of someone else as a backup while I am on vacation (if we want to have it). But, this bot will not change what we are already doing. If people don't review, the bot will not make them review.

#17 Updated by jorauch 11 months ago

  • Status changed from In Progress to Feedback

Putting to feedback to discuss this in the next groominh meeting

#18 Updated by mgriessmeier 11 months ago

  • Subject changed from [functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github to [functional][u] test code feedback - integrate bots to test distribution on github
  • Description updated (diff)
  • Status changed from Feedback to Workable
  • Target version changed from Milestone 23 to Milestone 24

removed timebox, see https://progress.opensuse.org/issues/47441#note-12 for documentation
updated milestone
put back to workable

#19 Updated by jorauch 11 months ago

  • Description updated (diff)

Updating according the results of the retro

#20 Updated by szarate 10 months ago

One of the Ideas that come to mind with this, is that maybe we could introduce the Stale Probot and mark the pr's as stale, closing after 3 months of inactivity or so.

#21 Updated by mgriessmeier 10 months ago

  • Target version changed from Milestone 24 to Milestone 25

#22 Updated by jorauch 8 months ago

  • Due date deleted (26/03/2019)

removing due date to stop endless mails

#23 Updated by mgriessmeier 6 months ago

  • Priority changed from Normal to High
  • Target version changed from Milestone 25 to Milestone 27

@jojo, @szarate - please check the status here

#24 Updated by mgriessmeier 5 months ago

  • Target version changed from Milestone 27 to Milestone 28

#25 Updated by SLindoMansilla 5 months ago

  • Description updated (diff)
  • Status changed from Workable to Resolved

#26 Updated by szarate 5 months ago

  • Description updated (diff)
  • Status changed from Resolved to Workable
  • Assignee deleted (jorauch)
  • Priority changed from High to Normal

#27 Updated by SLindoMansilla 5 months ago

  • Estimated time set to 42.00

#28 Updated by szarate 5 months ago

  • Description updated (diff)

I'm up for pairing with somebody that wants to work on this... should be fairly easy in a day or two

#29 Updated by SLindoMansilla 5 months ago

  • Description updated (diff)

#30 Updated by mgriessmeier about 1 month ago

  • Target version changed from Milestone 28 to Milestone 31

Also available in: Atom PDF