Add delay between pull request updates #278
Conversation
There was a problem hiding this comment.
While I think the approach of sleeping ahead of time works, it'd be nice to consider handling github rate limit errors and retrying with delays. I know the latter approach needs to handle state a bit differently from what we're doing today, so maybe its not a great use of time atm.
In projects with many open PRs using the automatic updates feature, Bulldozer can trigger GitHub's secondary rate limits by updating (and possibly querying) pull requests too quickly. To avoid this, add some delay between operations. For queries, use a fixed 250ms delay for now. This shouldn't slow things down too much, but does prevent Bulldozer from blasting through PRs as fast as possible. For updates, use an exponential delay that starts at 1s and maxes out at 60s. I believe the secondary limits (at least for GitHub Enterprise) are computed over a minute, so this will limit us to one update per minute for large batches. I don't think this will be prohibitively slow, but we could explore making this configurable in the future. As part of this, refactor the logic so that we don't block queries on previous updates.
bluekeyes
force-pushed
the
bkeyes/update-delay
branch
from
39bad1e
to
991e3f8
Compare
|
Based on @daiglebg's testing in #276, I've changed this to use a fixed 2 second delay for updates. With the maximum value that low, exponential backoff hit the limit so fast that the complexity didn't seem worthwhile. To compensate, I've added configuration options for both delays so that users can increase them or disable them if needed. The configuration is undocumented for now, as I think most people should use the defaults. |
|
@bluekeyes are you still interested in merging this? |
In projects with many open PRs using the automatic updates feature,
Bulldozer can trigger GitHub's secondary rate limits by updating (and
possibly querying) pull requests too quickly. To avoid this, add some
delay between operations.
For queries, use a fixed 250ms delay for now. This shouldn't slow things
down too much, but does prevent Bulldozer from blasting through PRs as
fast as possible.
For updates, use an exponential delay that starts at 1s and maxes out at
60s. I believe the secondary limits (at least for GitHub Enterprise) are
computed over a minute, so this will limit us to one update per minute
for large batches. I don't think this will be prohibitively slow, but we
could explore making this configurable in the future.
As part of this, refactor the logic so that we don't block queries on
previous updates.
This should help with the issues reported in #276.
Requires #277 to merge first.