Share your thoughts on InnerSource Metrics

[rrrutledge]

Answer the following:

• Why do you want to measure metrics on InnerSource?
• What specific metrics do you want to measure?
• Who will look at these metrics (and what actions will they take based on what they see)?

[dellagustin-sap]

There is some connection of this discussion with #31 (Why and how to execute an InnerSource survey?) - the results of the survey, are, to a certain extent, metrics.
Some of the main questions we have in our survey are:

• Do you know what InnerSource is?
• Have you ever contributed to InnerSource projects?
  There are more questions to drill down impediments and such, but the ones above can already trigger actions like more priority to awareness activities.

[rrrutledge]

Not a priority right now.

[rrrutledge]

Many important metrics come from non-automated sources like surveys.

[rrrutledge]

Why do you want to measure metrics on InnerSource?

• I want to show the successes of each individual project and the overall program.
• Find places where we could be having more success as an individual project or overall program.

What specific metrics do you want to measure?

• Count of reuse.
• Count of contribution.
• Ability to segment both of these based on business unit and project.

Who will look at these metrics (and what actions will they take based on what they see)?

• Executive leadership will look to determine the overall success of the program.
• The ISPO will look to improve participation where it is weak.
• Managers will look to determine success of individual projects.
• Project leaders will look to improve participation in their project.
• Project leaders will look to help consumers to stay upgraded to the latest project version.

Here is an internal page with similar information:

[jeffabailey]

Answer the following:

• Why do you want to measure metrics on InnerSource?

To communicate the value of InnerSource to management & senior leadership for the purpose of influencing hard investments in growth activities.

• What specific metrics do you want to measure?

We are measuring PR cycle time.

We want to measure:

1. Contribution Diversity – contributions made by departments, employee tenure, Etc.
2. PR Frequency
3. Contribution Business Value – the Business & Platform Capability a contribution is associated with. e.g. Customized Products -> Web-based 3D Rendering

• Who will look at these metrics (and what actions will they take based on what they see)?

Software Engineers, Product Managers, Engineering Managers, Eng & Product VPs

[rrrutledge]

Thanks for filling this out, @jeffabailey !

[rrrutledge]

For the folks that look at the metrics, what actions will they take based on what they see?

[jeffabailey]

1. Influence senior leadership to make InnerSource a core part of their team's daily workflow.
2. Demonstrate the tangible value of InnerSource to current & potential contributors.
3. Understand the overall growth, outreach, and value of InnerSource across the enterprise to define & drive strategic focus areas.

[rrrutledge]

@jeffabailey when you say value of InnerSource what is the value that you are looking for? Reading through what you've written here I feel like you're calling out the existence of InnerSource without being clear on what is the value to the company that you want to show?

[jeffabailey]

Here are the value outputs I have in mind.

1. Raw contributions that provide value to users.
2. Duplicate solution reduction.
3. Cross-team trust building & collaboration optimization.

[ghost]

Answer the following:

• Why do you want to measure metrics on InnerSource?
  Metrics are required to understand return on investment. Whether a business or personal venture, we are constantly evaluating whether "the juice is worth the squeeze". Even if the return is somewhat intangible (goodwill, reputation, learning), someone has to make a value decision based on whether the resources devoted to an activity support end goals. So as great as InnerSource patterns and values are, they are not the right tool for every job; metrics help us decide whether we need to use the tool better or get a different one altogether.
• What specific metrics do you want to measure?
  Cross-team collaboration (measured in PRs, comments, manager feedback, etc.)
  Software re-use
  Software quality
  Documentation (e.g. percentage increase in documented software and/or processes)
• Who will look at these metrics (and what actions will they take based on what they see)?
  ISPO and OSPO leads, upstream managers (including VPs and CTO-types), consultants/external stakeholders

[rrrutledge]

Thank you @joshclements-adi 🎉❗️

[rrrutledge]

For the folks that look at the metrics, what actions will they take based on what they see?

[rrrutledge]

@joshclements-adi what is the "juice" that you are looking for? Project reuse, quality, and documentation?

[ghost]

The "juice" is the benefit, the value, the why... from a business perspective, InnerSource processes should support business goals and thus the InnerSource metrics should be sub-sets of business metrics.

The metrics I mentioned (far from an all-inclusive list) would support efficiency metrics. Some folks might not think documentation supports efficiency, but I believe that documentation is an investment that reduces support inquiries and increases adoption (which allows an organization to get more life out of a product).

[zkoppert]

As an innersource project maintainer, the metrics I care about are time to first response & time to resolve on issues, PRs, and discussions. This may be very separate from the conversation about what an ISPO measures themselves on but thought I would throw it out there.

Why: in order to have high quality maintership and support for innersource
What: time to first response & time to resolve on issues, PRs, and discussions
Who: Innersource project maintainer and any project sponsors (engineering leadership wants to know key innersource projects are well maintained)

[rrrutledge]

Thanks, @zkoppert ❗️ Even though this audience and use case is different, it needs some of the same underlying data and so it is relevant to consider here. Could harvest data with the ability to build multiple views on top of it.

[JustinGOSSES]

We measure response time for cross-org PRs as part of "how well a repo is set up for collaboration".

[rrrutledge]

@zkoppert are those things that you measure today or things that you aspire to measure in the future?

[zkoppert]

@rrrutledge We are measuring those today

[JustinGOSSES]

Then numbers mentioned are the "specific metrics do you want to measure?" in section 2.

Why do you want to measure metrics on InnerSource?

• To have quantified evidence of internal usage and contribution across boundaries when someone says "no one does InnerSource or do we care about this as it is an edge case". (1,3,4,5)
• To be able to make good recommendations to others it helps to be able to identify different types of collaboration. For example: one-time small corrections, product-driven multi-team intense collaboration, internal service, guidance or template, etc. (1)
• To have information to identify why certain repositories may not be getting any cross team contributions (1,3,5,6)
• To identify and understand hotspots of collaboration. (1,6)
• To generate metadata for a better search experience so that users can opt-in to filter their search results to repositories more likely to be useful for others. (1,2,3,4,5)
• As input into other processes to identify sources of collaboration friction (2,3,4,5,6)
• To supply information on usage back to project originators so they can better represent the business value of their projects to their own management. (1,3,5)

What specific metrics do you want to measure?

1. Pull requests from someone not in the team of the majority of the contributors to a repository
2. InnerSource repo infrastructure:

• Existence of README.md
• Existence of CONTRIBUTING.md or a CONTRIBUTING section in the README
• Whether README comes from template, which one of known templates.
  • If an identified README was used, was it filled out beyond the description field at the top?

3. Internal reuse rates. *Currently, this does not exist but might in future so listing for completeness sake and to highlight its value.
4. Whether repository has been highlighted for potential reuse by placing it in an internal portal or listing.
5. Usage of locally-sourced open source. This refers to open source created by the same enterprise and used internally.
6. InnerSource PR response times.

Who will look at these metrics (and what actions will they take based on what they see)?

• Developers of a repository (3*, 4, 5, 6)
• Managers of developer & PM teams (1,2,3*,5, 6)
• OSPO/ISPO (1,2,3*,5)
• Infrastructure and internal tooling teams (1,3*,5)
• Upper management (1,3*,5)

It is worth mentioning that getting others to look at these metrics and change behaviors is a big challenge. Usage has been enough to make it worthwhile, but it is very spotty distribution. Never done a complete accounting of this, but I am now wondering if there might be more usage of constructed raw data products that get built into someone else's dashboard, calculations, etc. than in fully built out end user data applications?

[rrrutledge]

There are different platform limitations (e.g. API rate limit in an SCM vs. putting things in a backend where there is no rate limit).

Also different audiences and goals:

• Developer productivity metrics for management.
• Community health metrics for project maintainers.
• Project health metrics for potential consumers.

The same underlying data may feed into views for multiples of these scenarios.

[rrrutledge]

It is hard to develop dashboard for maintainers because they care about different things. You make something so general that nobody cares about it. A dashboard is something that people need to remember to go visit, which is a big hurdle. Dashboards built for a particular meeting/ceremony/process have been more successful. Integrating into existing workflows (e.g. VS code, PR bots, dev portal, etc.) can help.

[rrrutledge]

Thank you for this detailed listing and summary, @JustinGOSSES.

[rrrutledge]

To be able to make good recommendations to others it helps to be able to identify different types of collaboration

How does metric #1 help with this?

[rrrutledge]

@azinshahidi2019 mentions: From the business perspective want to consider ... will time-to-market accelerate by having InnerSource projects?

[rrrutledge]

It is important to come up with metrics that the business cares about, but it is hard to show improvement in a metric if the baseline isn't already being measured. In that case we need to both start measuring the baseline before we can show the improvement.

[rrrutledge]

This is hard to measure (correlation vs. causation).

[tfroment]

I will answer using the content I shared through my community talk in May (from https://www.youtube.com/watch?v=_2sK7hPITDU) - with slides at https://github.com/tfroment/public-presentations/raw/main/AdvocatingStrategicValueOfAnISPO.pptx

Why do you want to measure metrics on InnerSource?

Because if you want members of an Inner Source community (or an ISPO) and the Company to invest what you ask of them (arrow 1 and 2 in the previous diagram), THEN you must measure the value they will receive in return (arrow 3 and 4, respectively):

What specific metrics do you want to measure?
It heavily depends on the context, here are for me the key parameters

• (a) at Company (executive) level - then depends on impacted` business function
• (b) at Sponsor (Middle Management, internal sponsors) level
• (c) at individual level
  
  Tricky part here: some of the metrics are impossible to measure (green part) while some others are very concrete (black) but won't speak for some categories (e.g. obviously most of the technical-oriented metrics like PR and forks for instance won't be understood by exectives who speaks "language" of the money, so will be important to "translate" that data into "$$$". Grey part is most of the time somewhere in the middle and could be translated into "$$$" with some additionnal data analysis work

Who will look at these metrics (and what actions will they take based on what they see)?
For me there are 3 categories of actors:

• ISPO Members/Leads : the people who speak the language of the community, they are convinced about the rational, but speak the language of community (somewhat technical / are community enthousiasts)
• Sponsors : are most of the time Program/product/middle managers - They can help in the translation "community metrics" into "money language metrics"
• Executives / decision makers : who expect ROI for the company (in $$$)

[rrrutledge]

Thanks, @tfroment. I took a look at your slides and will watch the recording as well.

[rrrutledge]

I captured everything that I could glean from these comments onto this board. Take a look and make sure that your company is represented.

[dicortazar]

I'm a bit late to this discussion, but we published last year a paper together with Santander Bank on the first metrics to measure collaboration. This is available at [1]. I'd say the paper answers the first two questions at [2], and I'm not fully sure this answers the third question. (I can share more details on it if needed as this is not public access).

In addition to this, I see you seem to follow a GQM approach what is great! This was reference in the past in other cases as in the book Managing InnerSource Projects. There, there's a description of it at a high level [3] that could be useful to share here as well. This has been tested in the past with existing customers (wearing my Bitergia hat) and it is accepted by academia as a way to discover the metrics needed (wearing my researcher hat). At [4] there is another bunch of metrics following this approach.

Last but not least, there is a pattern discussing about metrics. This is called "Introducing metrics in InnerSource" [5].

Do you think this is useful for this conversation? I may try to fit all of this in the existing work if that makes sense.

[1] https://dl.acm.org/doi/abs/10.1145/3524842.3528044
[2] #60 (comment)
[3] https://innersourcecommons.gitbook.io/managing-inner-source-projects/introduction-1/gqm
[4] https://github.com/InnerSourceCommons/managing-inner-source-projects/blob/master/measuring/metrics.md
[5] https://github.com/InnerSourceCommons/InnerSourcePatterns/blob/main/patterns/1-initial/introducing-metrics-in-innersource.md

[rrrutledge]

Thanks for sharing @dicortazar ❗ I think that these things are relevant. I think we want all of this stuff to end up in one place (Managing InnerSource Projects?). Did you see this Miro board where we capture the metrics that folks in this working group are actually involved with?

[rrrutledge]

This would be a great addition to the Managing InnerSource Projects book.