[MDBF-572] Add time to first meaningful response to pull request metrics Created: 2023-06-20  Updated: 2023-07-05  Resolved: 2023-07-05

Status: Closed
Project: MariaDB Foundation Development
Component/s: None
Affects Version/s: None
Fix Version/s: N/A

Type: Task Priority: Major
Reporter: Andrew Hutchings Assignee: Andrew Hutchings
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: 0d
Time Spent: 3.75d
Original Estimate: Not Specified


 Description   

We need a "time to first meaningful response" metric. The current plan for this is to add code to the pull request metrics scripts which will:

1. Pull the comments for each pull request for the week being parsed
2. Look for the first comment from a staff member of MariaDB Corporation or Foundation (maybe using a list of members to MariaDB GitHub?), opening user and comment user cannot be the same person.
3. Work out the time difference between open and first comment that meets clause 2
4. Average this over the week.

Any that do not have a first meaningful response will not count towards the time, but will be logged using a counter, giving a second metric of number of pull requests with no meaningful response.

Code for this is in the pull-requests directory of https://github.com/MariaDB/metrics

 Comments   
Comment by Eric Herman [ 2023-06-20 ]

To me, this looks like a good place to start.

It is far beyond the scope of the initial version of the script to try differentiate "Hello Nurse" comments from truly helpful comments, thus I support starting with a plan like this.

Once implemented, it will be possible to review how good or bad it seems to be.

Maybe it will need tuning, yet maybe it will be good enough that the few not-very-meaningful responses will not change the weekly average enough to impact prioritization decisions.

Comment by Eric Herman [ 2023-06-21 ]

Related, some time ago, Mozilla published Measuring Engagement

If someone has a different source for measuring the relationship of responsiveness and return contributions, please link it.

Comment by Andrew Hutchings [ 2023-06-29 ]

Modified the metrics pull request code to do this. Sent first batch of results to Eric to inspect.

This is how it works with the current patch:

1. Get a list of all MariaDB organisation members*
2. For each week it digs into all the PRs opened that week
3. For each PR it gets the list of comments and reviews (they are separate API calls, despite being in the same place on the web)
4. It finds the first comment or review that is from a MariaDB organisation member who isn't the PR author
5. If found, it works out the number of days between the PR opening and that comment/review
6. If there has been no comment or review from a MariaDB organisation member, an counter is incremented for "no meaningful response"
7. The time to first meaningful response for all the new PRs of that week are averaged. Returned in days rounded to 1 decimal place.
8. Three columns are added to the CSV "Time to first response,Total responded,Total non-responded"

  • - Otto is currently a MariaDB organisation member, I have hard-coded it to remove him for this list.

"No meaningful response" is possible if a pull request is opened in error and the closed. So, is not always a bad thing. I'm considering a filter for issues that are closed with no comments or reviews.

Comment by Andrew Hutchings [ 2023-07-03 ]

Today's update:

Two extra categories added:

  • Self Merge No Review - PRs that have not been reviewed by a MariaDB
    member and then merged by the author
  • Self Close No Review - PRs that have been closed by the author with no
    review to them

In addition:

  • Added draft PR count
  • Draft PRs are not scanned for the first response and no review metrics
  • Improved the API backoff code
  • Cleanup CSV header
Comment by Andrew Hutchings [ 2023-07-05 ]

Added verbose mode to PR script to verify the results. Used this to verify several months of results.

Changes made:

  • Using argparse for the parsing of arguments
  • Add verbose option which shows details of data retrieved
  • Remove some duplicated code for retrieving data from URLs
Comment by Andrew Hutchings [ 2023-07-05 ]

Final changes and verify data

Generated at Thu Feb 08 03:38:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.