Finding the original code + contributors from the Github API

[Baptoshi]

Select Topic Area

Question

Body

I'm currently developing a service that requires identifying original code on GitHub (excluding forks and copy-cat) and the developers involved with the specific codebase.

We've been using the GitHub API, but we've encountered a couple of challenges:

• We are restricted by a query size limit, which doesn't allow more than 1000 characters.
• The API seems to flag special characters, such as parentheses (), which complicates our queries.

Could you suggest any workarounds or solutions to these issues?

Thanks for your help!

[samus-aran]

Hi @Baptoshi 👋

What GitHub API are you referring to? Rest or GraphQL?

[Le-Caignec]

Hey we are using Rest API through octokit !

[Baptoshi]

@samus-aran Yes we are using Rest API with octokit.

[Baptoshi]

@samus-aran any advice? :)

[samus-aran]

Hi there!

We are restricted by a query size limit, which doesn't allow more than 1000 characters.

We're not aware of a documented limit for query string length in the API. 🤔 What status code are you get back when you hit the limit?

The API seems to flag special characters, such as parentheses (), which complicates our queries.

Yes, standard rules apply for encoding query strings. Keys and values should be URL encoded. We don't think quotes are necessary, but it depends on the client that is being used.

[Le-Caignec]

Hey @samus-aran i'm using octokit sdk : this is an example of an issue that i have when i search through octokit a too long piece of code :

Failed to search GitHub for source code: RequestError [HttpError]: query exceeds max length - https://docs.github.com/rest/search/search#search-code
    at /Users/robin/Documents/Reputable/reputable-core/node_modules/@octokit/request/dist-node/index.js:124:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async requestWithGraphqlErrorHandling (/Users/robin/Documents/Reputable/reputable-core/node_modules/@octokit/plugin-retry/dist-node/index.js:71:20)
    at async Job.doExecute (/Users/robin/Documents/Reputable/reputable-core/node_modules/bottleneck/light.js:405:18) {
  status: 400,
  response: {
    url: 'https://api.github.com/search/code?q=%2F%2F%20SPDX-License-Identifier%3A%20GPL-2.0-or-later%0Apragma%20solidity%20%3D0.7.6%3B%0Apragma%20abicoder%20v2%3B%0A%0Aimport%20%27%40uniswap%2Fv3-core%2Fcontracts%2Flibraries%2FSafeCast.sol%27%3B%0Aimport%20%27%40uniswap%2Fv3-core%2Fcontracts%2Flibraries%2FTickMath.sol%27%3B%0Aimport%20%27%40uniswap%2Fv3-core%2Fcontracts%2Finterfaces%2FIUniswapV3Pool.sol%27%3B%0A%0Aimport%20%27.%2Finterfaces%2FISwapRouter.sol%27%3B%0Aimport%20%27.%2Fbase%2FPeripheryImmutableState.sol%27%3B%0Aimport%20%27.%2Fbase%2FPeripheryValidation.sol%27%3B%0Aimport%20%27.%2Fbase%2FPeripheryPaymentsWithFee.sol%27%3B%0Aimport%20%27.%2Fbase%2FMulticall.sol%27%3B%0Aimport%20%27.%2Fbase%2FSelfPermit.sol%27%3B%0Aimport%20%27.%2Flibraries%2FPath.sol%27%3B%0Aimport%20%27.%2Flibraries%2FPoolAddress.sol%27%3B%0Aimport%20%27.%2Flibraries%2FCallbackValidation.sol%27%3B%0Aimport%20%27.%2Finterfaces%2Fexternal%2FIWETH9.sol%27%3B%0A%0A%2F%2F%2F%20%40title%20Uniswap%20V3%20Swap%20Router%0A%2F%2F%2F%20%40notice%20Router%20for%20stateless%20execution%20of%20swaps%20against%20Uniswap%20V3%0Acontract%20SwapRouter%20is%0A%20%20%20%20ISwapRouter%2C%0A%20%20%20%20PeripheryImmutableState%2C%0A%20%20%20%20PeripheryValidation%2C%0A%20%20%20%20PeripheryPaymentsWithFee%2C%0A%20%20%20%20Multicall%2C%0A%20%20%20%20SelfPermit%0A%7B%0A%20%20%20%20usin+in%3Afile+extension%3Asol+filename%3ASwapRouter.sol+path%3Acontracts%2F&per_page=100',
    status: 400,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
      'cache-control': 'no-cache',
      'content-length': '115',
      'content-security-policy': "default-src 'none'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Mon, 13 May 2024 17:30:49 GMT',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'GitHub.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      vary: 'Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding, Accept, X-Requested-With',
      'x-accepted-oauth-scopes': '',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-api-version-selected': '2022-11-28',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': 'E41B:2D7D00:36B629FD:36F73776:66424E49',
      'x-oauth-scopes': 'admin:enterprise, admin:gpg_key, admin:org, admin:org_hook, admin:public_key, admin:repo_hook, admin:ssh_signing_key, audit_log, codespace, copilot, delete:packages, delete_repo, gist, notifications, project, repo, user, workflow, write:discussion, write:packages',
      'x-ratelimit-limit': '10',
      'x-ratelimit-remaining': '9',
      'x-ratelimit-reset': '1715621509',
      'x-ratelimit-resource': 'code_search',
      'x-ratelimit-used': '1',
      'x-xss-protection': '0'
    },
    data: {
      message: 'query exceeds max length',
      documentation_url: 'https://docs.github.com/rest/search/search#search-code'
    }
  },
  request: {
    method: 'GET',
    url: "https://api.github.com/search/code?q=%2F%2F%20SPDX-License-Identifier%3A%20GPL-2.0-or-later%0Apragma%20solidity%20%3D0.7.6%3B%0Apragma%20abicoder%20v2%3B%0A%0Aimport%20'%40uniswap%2Fv3-core%2Fcontracts%2Flibraries%2FSafeCast.sol'%3B%0Aimport%20'%40uniswap%2Fv3-core%2Fcontracts%2Flibraries%2FTickMath.sol'%3B%0Aimport%20'%40uniswap%2Fv3-core%2Fcontracts%2Finterfaces%2FIUniswapV3Pool.sol'%3B%0A%0Aimport%20'.%2Finterfaces%2FISwapRouter.sol'%3B%0Aimport%20'.%2Fbase%2FPeripheryImmutableState.sol'%3B%0Aimport%20'.%2Fbase%2FPeripheryValidation.sol'%3B%0Aimport%20'.%2Fbase%2FPeripheryPaymentsWithFee.sol'%3B%0Aimport%20'.%2Fbase%2FMulticall.sol'%3B%0Aimport%20'.%2Fbase%2FSelfPermit.sol'%3B%0Aimport%20'.%2Flibraries%2FPath.sol'%3B%0Aimport%20'.%2Flibraries%2FPoolAddress.sol'%3B%0Aimport%20'.%2Flibraries%2FCallbackValidation.sol'%3B%0Aimport%20'.%2Finterfaces%2Fexternal%2FIWETH9.sol'%3B%0A%0A%2F%2F%2F%20%40title%20Uniswap%20V3%20Swap%20Router%0A%2F%2F%2F%20%40notice%20Router%20for%20stateless%20execution%20of%20swaps%20against%20Uniswap%20V3%0Acontract%20SwapRouter%20is%0A%20%20%20%20ISwapRouter%2C%0A%20%20%20%20PeripheryImmutableState%2C%0A%20%20%20%20PeripheryValidation%2C%0A%20%20%20%20PeripheryPaymentsWithFee%2C%0A%20%20%20%20Multicall%2C%0A%20%20%20%20SelfPermit%0A%7B%0A%20%20%20%20usin+in%3Afile+extension%3Asol+filename%3ASwapRouter.sol+path%3Acontracts%2F&per_page=100",
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit.js/3.1.2 octokit-core.js/5.1.0 Node.js/18.20.1 (darwin; arm64)',
      authorization: 'token [REDACTED]'
    },
    request: { hook: [Function: bound bound register] }
  }
}

you can reproduce this issue, with the piece of code :

const octokit = new Octokit({
auth: YOUR_GITHUB_API_TOKEN,
})

let query =
    "// SPDX-License-Identifier: GPL-2.0-or-later\npragma solidity =0.7.6;\npragma abicoder v2;\n\nimport '@uniswap/v3-core/contracts/libraries/SafeCast.sol';\nimport '@uniswap/v3-core/contracts/libraries/TickMath.sol';\nimport '@uniswap/v3-core/contracts/interfaces/IUniswapV3Pool.sol';\n\nimport './interfaces/ISwapRouter.sol';\nimport './base/PeripheryImmutableState.sol';\nimport './base/PeripheryValidation.sol';\nimport './base/PeripheryPaymentsWithFee.sol';\nimport './base/Multicall.sol';\nimport './base/SelfPermit.sol';\nimport './libraries/Path.sol';\nimport './libraries/PoolAddress.sol';\nimport './libraries/CallbackValidation.sol';\nimport './interfaces/external/IWETH9.sol';\n\n/// @title Uniswap V3 Swap Router\n/// @notice Router for stateless execution of swaps against Uniswap V3\ncontract SwapRouter is\n    ISwapRouter,\n    PeripheryImmutableState,\n    PeripheryValidation,\n    PeripheryPaymentsWithFee,\n    Multicall,\n    SelfPermit\n{\n    usin+in:file+extension:sol+filename:SwapRouter.sol+path:contracts/";

const results = await octokit.paginate(octokit.rest.search.code, {
 q: query,
 per_page: 100, // Max allowed per page for this endpoint
});