action #47441
closed
[functional][u] test code feedback - integrate bots to test distribution on github
Added by szarate almost 6 years ago.
Updated over 4 years ago.
Target version:
SUSE QA (private) - Milestone 31
Description
Currently there's the problem with Pull requests sometimes being forgotten and open forever, leading to tickets that are open for too long just because a PR wasn't merged in time.
There's the proposal of using a bot that upon certain syntax either pings the person on irc, or better writes on an irc channel (#testing or #opensuse-factory? or #opensuse-someotherchannel? without being too noisy...
There are many bots that are available for this:
Requirements for an acceptable bot:¶
The bot shall be able to fulfill the Acceptance Criteria post timebox.
These are suggestions, but definitely not limited to those. A quick test should allow us to move forward and pick what's best for the team.
Suggestions¶
- Give a quick check to the options given or look for some alternatives
- Test the bot, ideally it will react when a PR has more than X days without progress, and when it is mentioned or when it reads a comment with certain syntax.
Tasks:¶
Acceptance Criteria: Post timebox¶
- AC.1: Bot is running and setup is documented in a wiki
- AC.2: PR older than 60 days without activity are considered stale
- AC.3: PR that is already marked as stale, with over 7 days of inactivity are closed
- AC.4: PR with labels RFC are excempt (can be expanded later)
- AC.5: Stale PR are marked with STALE label
Suggestions¶
To make it easier to test, it can be initially set up in the person's own github fork of os-autoinst-distri-opensuse (or any other repo) with very short times
- Target version changed from Milestone 22 to Milestone 24
Actually M22 is over. Let's focus a bit more on the existing tickets, when we trimmed down the backlog a bit we can pick this ok, shall we?
I don't understand how you see an 8h timebox + the ACs feasible. Did you confuse timebox and estimate? For a timebox I would expect as single AC that the findings are documented.
- Description updated (diff)
I don't understand how you see an 8h timebox + the ACs feasible. Did you confuse timebox and estimate? For a timebox I would expect as single AC that the findings are documented.
We agreed to this during the retrospective. I don't expect all of the AC's to be closed within one iteration, let's update the description :)
- Target version changed from Milestone 24 to Milestone 23
I see, that makes perfect sense. Thank you. So let's try in M23 because it is asked for by the team :)
- Due date set to 2019-03-26
- Status changed from New to Workable
- Related to action #48392: [functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule added
- Subject changed from [functional][u][y][timebox:8h] test code feedback - integrate bots to test distribution on github to [functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github
As QSF-u has picked it up first, let them handle it :)
- Status changed from Workable to In Progress
actually longer in progress
I will write a new one in Python, as the examples do not perfectly fit our needs
The way to go is:
g = GitHub instance
org = g.get_organization('os-autoinst')
repo = org.get_repo('os-autoinst-distri-opensuse')
pr = repo.get_pull(numberOfPR)
The bot is now able to connect to IRC and GitHub, it can collect the data of all PRs and detect PRs in IRC
Need to talk to szarate later about how this should work exactly since just printing the PR state to IRC would be quite a mess
As discussed with szarate:
- Bot shall scan PRs and consider them stalled by a preconfigured parameter (e.g. 3-5 days for OpenQA Engineers, 2 weeks for external guys)
- based on the List of old PRs we write a weekly comment, mentioning last commentator to GitHub (Bot Comments, and Author Comments do not count)
- we can print a list of the old PRs to an IRC channel (best packaged in a command, e.g. "/bot give PRs older than 5 days")
- Bot can print the age of mentioned PRs in a channel
Stuff can now be found here:
https://github.com/DrMullings/Scripts-Snippets-Stuff/tree/master/Marvin
- It is not yet capable of differentiating between internal and external guys (we should re-discuss this anyway, it's hard to do from a technical side)
- It can write comments on a PR after the staletime is over
- The list of PRs could be printed to the IRC, this is not yet implemented though
- It checks the age (of the last comment) of posted PRs
I still need write access ID and Secret for os-autoinst to get it productive
As spoken with jrauch,
I think that a bot writing comments in the PRs will cause a lot of noise. People who didn't care about reviewing them will ignore those emails, or even complain (like it happened with bugzilla tickets).
Writing reminder in an IRC chat is better, since only people in that chat will receive the messages. We need to define yet who is taking care of reviewing PR. I think that our old model "each one does when she/he has time" has been proven to not work. We need a dedicated group of people for that. And, if at the end, this person will be only me, the bot will cause only noise, since I will ignore it and continue with my approach:
- Look for tickets from U-Team on feedback
- If it has a PR, review (eventually merging it)
We need to think of someone else as a backup while I am on vacation (if we want to have it). But, this bot will not change what we are already doing. If people don't review, the bot will not make them review.
- Status changed from In Progress to Feedback
Putting to feedback to discuss this in the next groominh meeting
- Subject changed from [functional][u][timebox:8h] test code feedback - integrate bots to test distribution on github to [functional][u] test code feedback - integrate bots to test distribution on github
- Description updated (diff)
- Status changed from Feedback to Workable
- Target version changed from Milestone 23 to Milestone 24
- Description updated (diff)
Updating according the results of the retro
One of the Ideas that come to mind with this, is that maybe we could introduce the Stale Probot and mark the pr's as stale, closing after 3 months of inactivity or so.
- Target version changed from Milestone 24 to Milestone 25
- Due date deleted (
2019-03-26)
removing due date to stop endless mails
- Priority changed from Normal to High
- Target version changed from Milestone 25 to Milestone 27
@jojo, @szarate - please check the status here
- Target version changed from Milestone 27 to Milestone 28
- Description updated (diff)
- Status changed from Workable to Resolved
- Description updated (diff)
- Status changed from Resolved to Workable
- Assignee deleted (
jorauch)
- Priority changed from High to Normal
- Estimated time set to 42.00 h
- Description updated (diff)
I'm up for pairing with somebody that wants to work on this... should be fairly easy in a day or two
- Description updated (diff)
- Target version changed from Milestone 28 to Milestone 31
- Status changed from Workable to Feedback
- Assignee set to szarate
- Start date deleted (
2019-02-12)
- Status changed from Feedback to Resolved
Also available in: Atom
PDF