Project

General

Profile

action #121582

[tools][metrics] Calculate cycle + lead times for SUSE QE Tools continuously size:M

Added by okurz about 2 months ago. Updated about 3 hours ago.

Status:
Workable
Priority:
Normal
Assignee:
cdywan
Target version:
openQA Project - Ready
Start date:
2022-12-06
Due date:
% Done:

0%

Estimated time:

Description

Motivation

We collected cycle and lead times already in the past with some scripts. Then it was added to monitor.qa.suse.de but eventually stopped working and nobody resurrected it yet. LSG QE tracks metrics and tries to expand, see #118135 so we should contribute to that by bringing back cycle and lead time evaluations that can help us in our daily work.

Acceptance criteria

  • AC1: Up-to-date cycle and lead times for SUSE QE Tools can be found over monitor.qa.suse.de

Suggestions

Related issues

Related to openQA Tests - action #47891: [functional][u] Continuously update the QSF-u team charts (cycle time, etc.) to know how we performResolved2019-02-14

History

#2 Updated by cdywan about 2 months ago

  • Subject changed from [tools][metrics] Calculate cycle + lead times for SUSE QE Tools continuously to [tools][metrics] Calculate cycle + lead times for SUSE QE Tools continuously size:M
  • Description updated (diff)
  • Status changed from New to Workable

#3 Updated by cdywan about 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to cdywan

I'm taking a look. Not sure what existing scripts there might actually be, but I'll try to ask a couple more people to see what I can re-use.

#4 Updated by openqa_review about 2 months ago

  • Due date set to 2022-12-29

Setting due date based on mean cycle time of SUSE QE Tools

#5 Updated by tinita about 2 months ago

This project from Ivan might be helpful: https://github.com/ilausuch/redmine_statistics

#6 Updated by cdywan about 1 month ago

tinita wrote:

This project from Ivan might be helpful: https://github.com/ilausuch/redmine_statistics

Thanks for suggesting it. I also looked at stunning-octo-chainsaw (which measures numbers from GitHub if you couldn't guess from its name). Actually after taking a closer look and playing with it I realized we don't need to compute anything. Grafana will do that based on InfluxDB line protocol which periodically consumes total counts.

Output currently looks like so:

slo,team="QE Tools",title="Overall Backlog" count=91
slo,team="QE Tools",title="Workable Backlog" count=12
slo,team="QE Tools",title="Exceeding Due Date" count=0
slo,team="QE Tools",title="Untriaged QA" count=0
slo,team="QE Tools",title="Untriaged Tools Tagged" count=0
slo,team="QE Tools",title="SLO immediate (<1 day)" count=0
slo,team="QE Tools",title="SLO urgent (<1 week)" count=0
slo,team="QE Tools",title="SLO high (<1 month)" count=1
slo,team="QE Tools",title="SLO normal (<1 year)" count=0
slo,team="QE Tools",title="In Progress" count=5
slo,team="QE Tools",title="In Feedback" count=13

#7 Updated by cdywan about 1 month ago

  • Due date changed from 2022-12-29 to 2023-01-13
  • Status changed from In Progress to Feedback

Will wait on feedback for now, with all pieces in place. Feel free to comment on the sample output to confirm if this is what we want or if we want other numbers. Not going to work on it til next year, though.

#8 Updated by okurz about 1 month ago

I don't know how the above give cycle and lead times. Can you explain?

#9 Updated by cdywan about 1 month ago

okurz wrote:

I don't know how the above give cycle and lead times. Can you explain?

Let's take an example. slo,team="QE Tools",title="Workable Backlog" count=12 tells us that the Workable Backlog has 12 issues in it. This is updated hourly (configured in salt). I've not prepared Grafana dashboards yet that would render this into a graph.

I assume this is what we want. If not please explain by example what you might be expecting.

#10 Updated by tinita about 1 month ago

https://kanbanize.com/kanban-resources/kanban-software/kanban-lead-cycle-time

That's why I posted a link to Ivan's code, I think it might help

#11 Updated by okurz about 1 month ago

cdywan see the definition from the link that tinita posted

#12 Updated by okurz about 1 month ago

  • Due date changed from 2023-01-13 to 2023-01-20

christmas grace due date bump :)

#13 Updated by cdywan 28 days ago

Here's an example including additional fields analoguous to what's implemented in the redmine_statistics project. I was hoping to get some early feedback on the minimal approach, but I guess I'm keeping it in the same branch now:

slo,team="QE Tools",status="New",title="Overall Backlog" count=6 avg=4.231111111111111 med=2.4391666666666665 std=48.9769975
slo,team="QE Tools",status="Workable",title="Overall Backlog" count=10 avg=15.109333333333334 med=18.405833333333334 std=49.890591755829895
slo,team="QE Tools",status="In Progress",title="Overall Backlog" count=4 avg=3.396111111111111 med=3.2463888888888888 std=6.536649537037036
slo,team="QE Tools",status="Blocked",title="Overall Backlog" count=5 avg=11.00961111111111 med=7.204722222222222 std=91.1066115200617
slo,team="QE Tools",status="Workable",title="Workable Backlog" count=10 avg=15.109333333333334 med=18.405833333333334 std=49.890591755829895
slo,team="QE Tools",status="Feedback",title="In Feedback" count=13 avg=6.791153846153847 med=3.6266666666666665 std=69.11117296612852

#14 Updated by cdywan 19 days ago

  • Status changed from Feedback to In Progress

This is in progress.

#16 Updated by okurz 19 days ago

  • Related to action #47891: [functional][u] Continuously update the QSF-u team charts (cycle time, etc.) to know how we perform added

#17 Updated by okurz 19 days ago

I looked up the original tickets #43442 and #47891, according code repo https://github.com/DrMullings/Scripts-Snippets-Stuff

#18 Updated by okurz 17 days ago

As discussed "lead time" could be implemented by looking at "time_when_resolved - time_when_created", "cycle time" could be "time_when_resolved - time_when_assigned" which has the drawback that tickets that are assigned but not actively worked on also account for the cycle time but I consider those rare exceptions that maybe we can then avoid if the cycle time alerts us about those. An alternative is to sum up all times when ticket is in progress or feedback minus ticket in new or workable but I would leave that out for now.

#19 Updated by cdywan 17 days ago

Talked about it briefly. tinita raised the point that we should ideally measure each period of time a ticket is in progress, and scripts snippets stuff doesn't seem to handle that. I was taking a look at the journal data before, although it's not included in my branch so far, and I think that's doable.

#20 Updated by cdywan 17 days ago

okurz wrote:

As discussed "lead time" could be implemented by looking at "time_when_resolved - time_when_created", "cycle time" could be "time_when_resolved - time_when_assigned" which has the drawback that tickets that are assigned but not actively worked on also account for the cycle time

Right. That includes the user story "Kim is assigning themself to a ticket a few days before actively working on it".

#21 Updated by cdywan 10 days ago

  • Due date changed from 2023-01-20 to 2023-01-27
  • Status changed from In Progress to Workable

I'm not blocked here, but simply couldn't make time to work on the last step because of other tasks. The integration within Grafana also still needs to be tested, so even then we'd want to allow people to verify that we see according data on the dashboard.

Maybe I should for now make it Workable in case somebody else has spare cycles and is interested in working on it. Otherwise I plan to pick it up again next week.

#22 Updated by cdywan 4 days ago

  • Tags deleted (mob)

To figure out how the data needs to look Tina and I took an example and analyzed it from the raw data to the query used in a Grafana dashboard:

Transferring this to the context of cycle time we want 1) the number of resolved tickets and 2) the sum of time those tickets spent "in progress", calculated in the backlogger code. An example of multiple days, assuming we feed data daily and look at the data from the last day, could look like so:

day1: count_resolved=1 sum_cycle_time=14
day2: count_resolved=0 sum_cycle_time=0 (no data)
day3: count_resolved=2 sum_cycle_time=10

Grafana will process this data and make available the average/median/whatever we choose in a query for a given time span.

#23 Updated by cdywan about 3 hours ago

  • Due date deleted (2023-01-27)

Due date is not generally applicable to workable

Also available in: Atom PDF