🚀 GitHub Copilot Metrics API now available in public beta

[Akash1134]

We’re excited to announce the public beta of the GitHub Copilot Metrics API, available to all customers of GitHub Copilot Business and GitHub Copilot Enterprise starting today.

The GitHub Copilot Metrics API is designed to supply you with information about Copilot’s usage within your organizations. The data from the API is intended to be consumed and combined with your organization’s own data to create greater visibility into how Copilot fits into the bigger picture of your software development cycle.

To know what functionality will be included in the Public Beta, please refer the changelog post.

Documentation and Resources

To help you get started, we have prepared comprehensive documentation and resources:

• 📝 API Documentation: Explore the detailed API documentation, including metrics definitions here.
• 📊 Visualization: You can find an open source tool for consuming and visualizing data from the endpoint here
• 🧑‍🎓 Learning Pathway: You can find an extended article on measuring the impact of GitHub Copilot here

Participate in the Public Beta!

Your feedback during this beta phase is invaluable to us. We encourage you to share your experiences, which will be instrumental in refining and enhancing the API as we look toward the future.

Stay tuned for updates and enhancements throughout the beta period. We’re committed to delivering a robust and feature-rich API that meets your needs and expectations.

[aline-viana]

Hi, we are very excited to test this resourse!
I made an api call using javascript to get information, but I receive the status: 404 response.
So, for some company this feature is not available yet?
I try org and enterprise and the telemetry is active

[gmotodd]

Same case for me, calling the API through Postman and using fine-grained tokens on both, Orgs/{} and Orgs/{}/Teams{} URIs, return HTTP-404 as if our Org and Team(s) did not exist.

I validated this by purposedly passing correct/incorrect URI elements ("org" instead of "orgs") to see whether the URI is valid, or my particular resources (Org and Team) do not exist, and I did get different responses.

When calling using "org" only, which is wrong, I get a 404 pointing me to the generic API documentation:
{
"message": "Not Found",
"documentation_url": "https://docs.github.com/rest"
}

And when sending the correct word "orgs", I get a 404 pointing me to this specific API's documentation:
{
"message": "Not Found",
"documentation_url": "https://docs.github.com/rest/copilot/copilot-usage#get-a-summary-of-copilot-usage-for-organization-members"
}

This tells me that our Org and Team(s) resources are just not found.

Another post in this forum below (https://github.com/orgs/community/discussions/112377) seems to indicate that the API worked for them.

Does anybody know if something else needs to be enabled Admin-wise in the github.com portal for the API to find our resources? We have been using GH Copilot for 6+ months now.

[SIkebe]

calling the API through Postman and using fine-grained tokens

How about PAT (classic)? Fine-grained personal access tokens is not supported according to the docs.

This endpoint does not work with GitHub App user access tokens, GitHub App installation access tokens, or fine-grained personal access tokens.
https://docs.github.com/en/rest/copilot/copilot-usage?apiVersion=2022-11-28#get-a-summary-of-copilot-usage-for-enterprise-members--fine-grained-access-tokens

[gmotodd]

Thanks for the thought. That is true for the Enterprises/{} URI. I am trying to consume the Orgs/{} and Orgs/{}/Teams{} URIs, for which the documentation does say that fine-grained personal access tokens are allowed.

And if the token was an issue, shouldn't I receive either 401 or 403 errors instead of 404 - resource not found?

[erikvanlith]

Hi, I also have the same experience.

• I tried with "Fine-grained personal access tokens". During the setup I added enough permissions and selected the organization. I get:

  "message": "Resource not accessible by personal access token",
  "documentation_url": "https://docs.github.com/rest/copilot/copilot-usage#get-a-summary-of-copilot-usage-for-organization-members"

• I also tried a classic token and then I get

  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest/copilot/copilot-usage#get-a-summary-of-copilot-usage-for-organization-members"

Are we missing something? We have a GitHub Copilot Business account.

[erikvanlith]

I think we found it and it is actually stated in the documentation (I have to admit ;) ): "Copilot Business or Copilot Enterprise organization owners, and owners and billing managers of their parent enterprises, can view Copilot usage metrics.". I personally have a github account and I am a member of the organization.

With the organization account you can make someone member, but also owner. For this to work it seems you have to be an owner. We will try some things, but I hope this might already help some people.

[gmotodd]

I also am an Enterprise and Org owner (not sure if you can get fine-grained tokens otherwise).
For kicks, I updated all my Account and Repository permissions to at least Read-Only, but to no avail still.

And again, if permissions are the issue, wouldn't the API responses be 401 (Unauthorized) or 403 (Forbidden), instead of 404 (Resource not found)?

Question: What is the difference between "PAT (classic)" tokens and fine-grained? And how do you obtain classic ones to give it a try?

[gmotodd]

Update: I tried calling the teams API (https://api.github.com/orgs/.../teams) with the same fine-grained token, and I did get an expected 403 saying that my token is not supported:

But not getting the same response in the Copilot endpoint kind of validates that the token is not the problem, but rather the resource value I'm passing that it is not found (I triple-checked syntax), or something else need to be enabled inside GitHub.

[gmotodd]

And when calling the Orgs endpoint (https://api.github.com/orgs/{...}) with the same fine-grained token, it returns data.

I'm assuming this validates that the org resource value I'm passing is valid, just not for the Copilot metrics endpoint. Would this be a sign that something else needs to be enabled/configured in the GitHub portal admin-wise?

[gmotodd]

Solved! I tried passing a Personal Access Token (Classic) and it worked. Go figure!

[aline-viana]

@gmotodd I try with classic token and work for me
https://github.com/settings/personal-access-tokens/new click in Token (classic) and generate new

[ndemayo-bah]

Hello! Seeing something odd maybe in the metrics being reported... it looks like the breakdown portion of the reponse is returning duplicated results. First screenshot is the data converted to csv, second shows the raw response from the endpoint... is this something others have noticed?

[ndemayo-bah]

Looking a bit closner, it does appear as though it hasn't done this since 04/18, so maybe it's something that has been fixed already.

[DevOps-zhuang]

Team, for the copilot metric API, My customer found two issues, and I can reproduce it in my environment, we both use Personal Access Token (Classic) 1) when calling organization level API like https://api.github.com/orgs/{MyOrg}/copilot/usage, it didn't return data on "day": "2024-04-05, the date before or after it can be returned, it is some interesting 2）the team level API(https://api.github.com/orgs/{org}/team/{team_slug}/copilot/usage) just returns empty, the http status code is 200, so it seems the authentication is right. and I am sure the team is over 20 people. are those of known issues? thanks.

[Akash1134]

We’ve updated how we calculate Last Activity to give you better clarity and are pausing access to the Team endpoint in the Metrics API.

Ahead of the GitHub Copilot Metrics API launch, we made an update to how we calculate Last Activity in order to provide more useful information for admins.

To know what has changed, please read this changelog

You can provide feedback and follow along for updates via this Discussion.