Question: secondary rate limits

[ritchie46]

Great project! It helped me a lot for one changelog generation.

Sadly, I keep hitting secondary rate limits. I've set up an authentication token, but that doesn't help. Any ideas?

Here are the logs:

docker run -it --env-file .env --rm -v "$(pwd)":/usr/local/src/your-app githubchangeloggenerator/github-changelog-generator --no-issues --no-unreleased -u pola-rs -p polarsUsing these options:
:date_format=>"%Y-%m-%d"
:output=>"AUTO_CHANGELOG.md"
:base=>"py-polars/CHANGELOG.md"
:issues=>false
:add_issues_wo_labels=>true
:add_pr_wo_labels=>true
:pulls=>true
:filter_issues_by_milestone=>true
:issues_of_open_milestones=>true
:author=>true
:unreleased=>false
:unreleased_label=>"Unreleased"
:compare_link=>true
:exclude_labels=>["no-changelog", "question"]
:summary_labels=>["Release summary", "release-summary", "Summary", "summary"]
:breaking_labels=>["backwards-incompatible", "Backwards incompatible", "breaking"]
:enhancement_labels=>["enhancement", "Enhancement", "Type: Enhancement"]
:bug_labels=>["bug", "Bug", "Type: Bug"]
:deprecated_labels=>["deprecated", "Deprecated", "Type: Deprecated"]
:removed_labels=>["removed", "Removed", "Type: Removed"]
:security_labels=>["security", "Security", "Type: Security"]
:configure_sections=>{}
:add_sections=>"{\"features\":{\"prefix\":\"**Enhancements:**\",\"labels\":[\"enhancement\"]}, \"documentation\":{\"prefix\":\"**Documentation updates:**\",\"labels\":[\"documentation\"]}, \"testing\":{\"prefix\":\"**Testing updates:**\",\"labels\":[\"testing\"]}}"
:issue_line_labels=>[]
:max_issues=>nil
:simple_list=>false
:ssl_ca_file=>nil
:verbose=>true
:header=>"# Changelog"
:merge_prefix=>"**Merged pull requests:**"
:issue_prefix=>"**Closed issues:**"
:summary_prefix=>""
:breaking_prefix=>"**Breaking changes:**"
:enhancement_prefix=>"**Implemented enhancements:**"
:bug_prefix=>"**Fixed bugs:**"
:deprecated_prefix=>"**Deprecated:**"
:removed_prefix=>"**Removed:**"
:security_prefix=>"**Security fixes:**"
:http_cache=>true
:require=>[]
:config_file=>".github_changelog_generator"
:user=>"pola-rs"
:project=>"polars"
:since_tag=>"py-polars-v0.10.1"
:token=>"hidden value"

WARNING: `Faraday::Connection#authorization` is deprecated; it will be removed in version 2.0.
While initializing your connection, use `#request(:authorization, ...)` instead.
See https://lostisland.github.io/faraday/middleware/authentication for more usage info.
Found 161 tags                                                         
Fetching tags dates: 161/161
Sorting tags...
Received issues: 1443                                                  
Pull Request count: 900                                                
Filtered pull requests: 864
Octokit::Forbidden: GET https://api.github.com/repos/pola-rs/polars/issues/181/events?per_page=100: 403 - You have exceeded a secondary rate limit. Please wait a few minutes before you try again. // See: https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits
Exceeded retry limit

[webrails]

The same applies for me. While fetching events for hundreds of issues, the secondary rate limits are exceeded:

Octokit::Forbidden: GET https://api.github.com/repos/xxx/yyy/issues/500/events?per_page=100: 403 - You have exceeded a secondary rate limit. Please wait a few minutes before you try again. // See: https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits

Is there a way to deal with this issue?

[sbrunner]

I'm also affected, Is it possible to limit the number of query?
Parameters like --since-tag don't look to affect the number of query...
Or be able to build the cache by chunk...

[bensheldon]

Maybe I got lucky, but it seemed like running the command multiple times with progressively larger --max-issues=200 (and then --max-issues=500, etc.) and then finally removing it overcame the secondary rate-limiting blocker. My hypothesis is that it incrementally filled the cached responses, but it could have simply been luck or a hiccup with GitHub's rate limiter.

My project recently hit 1,000+ PRs and Issues, so I imagine at best I've only bought myself more time with this tool as my number of PRs alone increase.

[aaronlippold]

Octokit::TooManyRequests GET https://api.github.com/repos/mitre/heimdall2/issues/4910/events?per_page=100: 403 - You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID FA2D:87B1:8879E3:1176C87:6570B7E6. // See: https://docs.github.com/en/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits; sleeping for 3348s...
Octokit::TooManyRequests GET https://api.github.com/repos/mitre/heimdall2/issues/4871/events?per_page=100: 403 - You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID FA2D:87B1:887A47:1176D5B:6570B7E6. // See: https://docs.github.com/en/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits; sleeping for 3348s...
Octokit::TooManyRequests GET https://api.github.com/repos/mitre/heimdall2/issues/4876/events?per_page=100: 403 - You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID FA2D:87B1:887A46:1176D4F:6570B7E6. // See: https://docs.github.com/en/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits; sleeping for 3348s...
Octokit::TooManyRequests GET https://api.github.com/repos/mitre/heimdall2/issues/4883/events?per_page=100: 403 - You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID FA2D:87B1:887A26:1176D0F:6570B7E6. // See: https://docs.github.com/en/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits; sleeping for 3348s...```

[aaronlippold]

This seems to be coming up again

[jimbali]

It never stopped happening for me. But if I just wait for an hour or so then it eventually works. 😑