Project

General

Profile

Actions

action #47441

closed

[functional][u] test code feedback - integrate bots to test distribution on github

Added by szarate about 5 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
SUSE QA - Milestone 31
Start date:
Due date:
% Done:

0%

Estimated time:
42.00 h

Description

Currently there's the problem with Pull requests sometimes being forgotten and open forever, leading to tickets that are open for too long just because a PR wasn't merged in time.

There's the proposal of using a bot that upon certain syntax either pings the person on irc, or better writes on an irc channel (#testing or #opensuse-factory? or #opensuse-someotherchannel? without being too noisy...

There are many bots that are available for this:

Requirements for an acceptable bot:

The bot shall be able to fulfill the Acceptance Criteria post timebox.

These are suggestions, but definitely not limited to those. A quick test should allow us to move forward and pick what's best for the team.

Suggestions

  • Give a quick check to the options given or look for some alternatives
  • Test the bot, ideally it will react when a PR has more than X days without progress, and when it is mentioned or when it reads a comment with certain syntax.

Tasks:

Acceptance Criteria: Post timebox

  • AC.1: Bot is running and setup is documented in a wiki
  • AC.2: PR older than 60 days without activity are considered stale
  • AC.3: PR that is already marked as stale, with over 7 days of inactivity are closed
  • AC.4: PR with labels RFC are excempt (can be expanded later)
  • AC.5: Stale PR are marked with STALE label

Suggestions

To make it easier to test, it can be initially set up in the person's own github fork of os-autoinst-distri-opensuse (or any other repo) with very short times


Related issues 1 (0 open1 closed)

Related to openQA Project - action #48392: [functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test scheduleResolvedybonatakis2019-02-252019-03-12

Actions
Actions #1

Updated by okurz about 5 years ago

  • Target version changed from Milestone 22 to Milestone 24

Actually M22 is over. Let's focus a bit more on the existing tickets, when we trimmed down the backlog a bit we can pick this ok, shall we?

I don't understand how you see an 8h timebox + the ACs feasible. Did you confuse timebox and estimate? For a timebox I would expect as single AC that the findings are documented.

Actions #2

Updated by szarate about 5 years ago

  • Description updated (diff)

I don't understand how you see an 8h timebox + the ACs feasible. Did you confuse timebox and estimate? For a timebox I would expect as single AC that the findings are documented.

We agreed to this during the retrospective. I don't expect all of the AC's to be closed within one iteration, let's update the description :)

Actions #3

Updated by okurz about 5 years ago

  • Target version changed from Milestone 24 to Milestone 23

I see, that makes perfect sense. Thank you. So let's try in M23 because it is asked for by the team :)

Actions #4

Updated by okurz about 5 years ago

  • Due date set to 2019-03-26
Actions #5

Updated by szarate about 5 years ago

  • Status changed from New to Workable
Actions #6

Updated by okurz about 5 years ago

  • Related to action #48392: [functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule added
Actions #7

Updated by jorauch about 5 years ago

  • Assignee set to jorauch

That looks lovely

Actions #8

Updated by okurz about 5 years ago

  • Subject changed from [functional][u][y][timebox:8h] test code feedback - integrate bots to test distribution on github to [functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github

As QSF-u has picked it up first, let them handle it :)

Actions #9

Updated by jorauch about 5 years ago

This one also looks useful:
https://github.com/probot/stale

Actions #10

Updated by jorauch about 5 years ago

  • Status changed from Workable to In Progress

actually longer in progress
I will write a new one in Python, as the examples do not perfectly fit our needs

Actions #11

Updated by jorauch about 5 years ago

The way to go is:

g = GitHub instance
org = g.get_organization('os-autoinst')
repo = org.get_repo('os-autoinst-distri-opensuse')
pr = repo.get_pull(numberOfPR)

Actions #12

Updated by jorauch about 5 years ago

Repo with the script:
https://github.com/DrMullings/Scripts-Snippets-Stuff/blob/master/Marvin/marvin.py

Progress so far:

  • able to age of all PRs
  • able to get age of comment of all PRs
  • able to get if a PR is approved
Actions #13

Updated by jorauch about 5 years ago

The bot is now able to connect to IRC and GitHub, it can collect the data of all PRs and detect PRs in IRC
Need to talk to szarate later about how this should work exactly since just printing the PR state to IRC would be quite a mess

Actions #14

Updated by jorauch about 5 years ago

As discussed with szarate:

  • Bot shall scan PRs and consider them stalled by a preconfigured parameter (e.g. 3-5 days for OpenQA Engineers, 2 weeks for external guys)
  • based on the List of old PRs we write a weekly comment, mentioning last commentator to GitHub (Bot Comments, and Author Comments do not count)
  • we can print a list of the old PRs to an IRC channel (best packaged in a command, e.g. "/bot give PRs older than 5 days")
  • Bot can print the age of mentioned PRs in a channel
Actions #15

Updated by jorauch about 5 years ago

Stuff can now be found here:
https://github.com/DrMullings/Scripts-Snippets-Stuff/tree/master/Marvin

  • It is not yet capable of differentiating between internal and external guys (we should re-discuss this anyway, it's hard to do from a technical side)
  • It can write comments on a PR after the staletime is over
  • The list of PRs could be printed to the IRC, this is not yet implemented though
  • It checks the age (of the last comment) of posted PRs

I still need write access ID and Secret for os-autoinst to get it productive

Actions #16

Updated by SLindoMansilla about 5 years ago

As spoken with jrauch,

I think that a bot writing comments in the PRs will cause a lot of noise. People who didn't care about reviewing them will ignore those emails, or even complain (like it happened with bugzilla tickets).

Writing reminder in an IRC chat is better, since only people in that chat will receive the messages. We need to define yet who is taking care of reviewing PR. I think that our old model "each one does when she/he has time" has been proven to not work. We need a dedicated group of people for that. And, if at the end, this person will be only me, the bot will cause only noise, since I will ignore it and continue with my approach:

  1. Look for tickets from U-Team on feedback
  2. If it has a PR, review (eventually merging it)

We need to think of someone else as a backup while I am on vacation (if we want to have it). But, this bot will not change what we are already doing. If people don't review, the bot will not make them review.

Actions #17

Updated by jorauch about 5 years ago

  • Status changed from In Progress to Feedback

Putting to feedback to discuss this in the next groominh meeting

Actions #18

Updated by mgriessmeier about 5 years ago

  • Subject changed from [functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github to [functional][u] test code feedback - integrate bots to test distribution on github
  • Description updated (diff)
  • Status changed from Feedback to Workable
  • Target version changed from Milestone 23 to Milestone 24

removed timebox, see https://progress.opensuse.org/issues/47441#note-12 for documentation
updated milestone
put back to workable

Actions #19

Updated by jorauch about 5 years ago

  • Description updated (diff)

Updating according the results of the retro

Actions #20

Updated by szarate almost 5 years ago

One of the Ideas that come to mind with this, is that maybe we could introduce the Stale Probot and mark the pr's as stale, closing after 3 months of inactivity or so.

Actions #21

Updated by mgriessmeier almost 5 years ago

  • Target version changed from Milestone 24 to Milestone 25
Actions #22

Updated by jorauch almost 5 years ago

  • Due date deleted (2019-03-26)

removing due date to stop endless mails

Actions #23

Updated by mgriessmeier over 4 years ago

  • Priority changed from Normal to High
  • Target version changed from Milestone 25 to Milestone 27

@jojo, @szarate - please check the status here

Actions #24

Updated by mgriessmeier over 4 years ago

  • Target version changed from Milestone 27 to Milestone 28
Actions #25

Updated by SLindoMansilla over 4 years ago

  • Description updated (diff)
  • Status changed from Workable to Resolved
Actions #26

Updated by szarate over 4 years ago

  • Description updated (diff)
  • Status changed from Resolved to Workable
  • Assignee deleted (jorauch)
  • Priority changed from High to Normal
Actions #27

Updated by SLindoMansilla over 4 years ago

  • Estimated time set to 42.00 h
Actions #28

Updated by szarate over 4 years ago

  • Description updated (diff)

I'm up for pairing with somebody that wants to work on this... should be fairly easy in a day or two

Actions #29

Updated by SLindoMansilla over 4 years ago

  • Description updated (diff)
Actions #30

Updated by mgriessmeier over 4 years ago

  • Target version changed from Milestone 28 to Milestone 31
Actions #31

Updated by okurz about 4 years ago

By now there are also more ideas based on github actions that can achieve something similar, e.g. https://github.com/actions/stale , similar to https://probot.github.io/apps/stale/ . I personally think that marking PRs as stale and eventually closing can be ok if done after a rather long time, e.g. months.

Actions #32

Updated by szarate almost 4 years ago

  • Status changed from Workable to Feedback
  • Assignee set to szarate
  • Start date deleted (2019-02-12)
Actions #34

Updated by szarate almost 4 years ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF