You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ampkeegan opened this issue
May 23, 2024
· 0 comments
Assignees
Labels
api: cloudchannelIssues related to the Channel Services API.status: investigatingThe issue is under investigation, which is determined to be non-trivial.
Per recommendation of support case GCP #51120117 I'm opening a ticket here. Not sure if this is a bug report, feature request, or I'm just doing something wrong, but it appears the basic Retry() functionality for google-cloud-channel (and maybe others) only works if the limit is reached when the initial request is made. If a limit is reached while paging through the response iterator, Retry() is not implemented.
Example code:
_RETRYABLE_REASONS = frozenset(
[StatusCode.RESOURCE_EXHAUSTED, ]
)
_UNSTRUCTURED_RETRYABLE_TYPES = (
ConnectionError,
exceptions.Aborted,
exceptions.DeadlineExceeded,
exceptions.InternalServerError,
exceptions.ResourceExhausted,
exceptions.ServiceUnavailable,
exceptions.Unknown,
exceptions.Cancelled,
exceptions.TooManyRequests
)
def _should_retry(exc):
if not hasattr(exc, "errors") or len(exc.errors) == 0:
# Check for unstructured error returns, e.g. from GFE
return isinstance(exc, _UNSTRUCTURED_RETRYABLE_TYPES)
reason = exc.errors[0].code()
return reason in _RETRYABLE_REASONS
custom_retry = Retry(
initial=1, # seconds (default: 0.1)
maximum=90.0, # seconds (default: 60.0)
multiplier=5, # default: 1.3
deadline=300.0, # seconds (default: 60.0)
predicate=_should_retry,
)
request = channel_v1.ListCustomersRequest(parent="accounts/" + channelAccount, page_size=50)
response = client_v1.list_customers(request, retry=custom_retry)
c = 0
for x in response.pages:
print(c)
c += 1
Running three simultaneous executions of this script will result in a RESOURCE_EXHAUSTED error. Immediately running a single instance of the script (while still within the window to be rate limited) will properly implement the Retry() with exponential backoff until the limit is removed. Once it can progress, it will fail again shortly while paginating through response.pages with another RESOURCE_EXHAUSTED.
The text was updated successfully, but these errors were encountered:
api: cloudchannelIssues related to the Channel Services API.status: investigatingThe issue is under investigation, which is determined to be non-trivial.
Per recommendation of support case GCP #51120117 I'm opening a ticket here. Not sure if this is a bug report, feature request, or I'm just doing something wrong, but it appears the basic Retry() functionality for google-cloud-channel (and maybe others) only works if the limit is reached when the initial request is made. If a limit is reached while paging through the response iterator, Retry() is not implemented.
Example code:
Running three simultaneous executions of this script will result in a RESOURCE_EXHAUSTED error. Immediately running a single instance of the script (while still within the window to be rate limited) will properly implement the Retry() with exponential backoff until the limit is removed. Once it can progress, it will fail again shortly while paginating through
response.pages
with another RESOURCE_EXHAUSTED.The text was updated successfully, but these errors were encountered: