Group updates failing to push to Discourse

[rinakondur]

My worker.log file is filling up with errors regarding group updates failing to push to Discourse. Not sure if something changed with a Discourse update or what but all of my information in the configs are correct. Wondering if anyone else is running into this or where I can look to troubleshoot this further.

Before and after this particular error is just logs of it trying to push and fail, stating it will retry in a certain amount of time before it just cancels the task with this error.

[2018-05-21 00:50:42,716: ERROR/ForkPoolWorker-1] Task discourse.update_groups[14772148-78bf-4eb3-9bf8-bcac4d07df94] raised unexpected: MaxRetriesExceededError("Can't retry discourse.update_groups[14772148-78bf-4eb3-9bf8-bcac4d07df94] args:(76,) kwargs:{}",)
Traceback (most recent call last):
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/allianceauth/services/modules/discourse/manager.py", line 176, in __exc
    raise DiscourseError(endpoint, r.json()['errors'])
allianceauth.services.modules.discourse.manager.DiscourseError: API execution failed.
Errors: ['You’ve performed this action too many times. Please wait 26 seconds before trying again.']
Endpoint: {'path': '/users/by-external/%s.json', 'method': 'get', 'args': {'required': [], 'optional': []}, 'parsed_url': '/users/by-external/76.json'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/allianceauth/services/modules/discourse/manager.py", line 185, in __exc
    r.raise_for_status()
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://forums.origindot.space/users/by-external/76.json?api_key=03ab09ba868ccc513bf4d453fb56d24990eaca36a092f7fa5d2e56b512df95a1&api_username=rinakondur

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/allianceauth/services/modules/discourse/tasks.py", line 49, in update_groups
    DiscourseManager.update_groups(user)
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/allianceauth/services/modules/discourse/manager.py", line 354, in update_groups
    username = DiscourseManager.__get_user_by_external(user.pk)['user']['username']
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/allianceauth/services/modules/discourse/manager.py", line 318, in __get_user_by_external
    return DiscourseManager.__exc(endpoint, u_id)
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/allianceauth/services/modules/discourse/manager.py", line 187, in __exc
    raise DiscourseError(endpoint, e.response.status_code)
allianceauth.services.modules.discourse.manager.DiscourseError: API execution failed.
Errors: 429
Endpoint: {'path': '/users/by-external/%s.json', 'method': 'get', 'args': {'required': [], 'optional': []}, 'parsed_url': '/users/by-external/76.json'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/celery_once/tasks.py", line 67, in __call__
    return super(QueueOnce, self).__call__(*args, **kwargs)
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/allianceauth/services/modules/discourse/tasks.py", line 52, in update_groups
    raise self.retry(countdown=60 * 10)
  File "/home/allianceserver/venv/auth/lib/python3.6/site-packages/celery/app/task.py", line 672, in retry
    self.name, request.id, S.args, S.kwargs))
celery.exceptions.MaxRetriesExceededError: Can't retry discourse.update_groups[14772148-78bf-4eb3-9bf8-bcac4d07df94] args:(76,) kwargs:{}

[rinakondur]

I should add that it APPEARS to work if a user is in a group that has already sync'd with Discourse but I've noticed this after a new corporation joined the alliance and their corporation group has not been pushed at all to Discourse. Even adding myself to the group in auth does not trigger it, just gets caught in an endless loop of trying and failing to update.

[Adarnof]

Oh boy. Discourse rate limits. I thought I'd never have to deal with this.

You can allegedly disable API rate limiting by setting DISCOURSE_MAX_ADMIN_API_REQS_PER_KEY_PER_MINUTE to a really big number in your app.yml env section. I'm really surprised you're hitting them at all - how many users do you have on Discourse?

This seems like another obvious candidate to employ celery bootsteps like I'd looked into for #1064 - I'll keep digging.

[rinakondur]

Maybe related, maybe not, but in the Notifications in auth I've been getting these repeatedly. Perhaps it's just repeatedly failing to update groups and keeps trying for some reason? We only have 110-120 users on Discourse at the moment so it's hard to imagine it's the API limit due to normal usage.

Discourse execution failed. Endpoint: {'path': '/users/by-external/%s.json', 'method': 'get', 'args': {'required': [], 'optional': []}, 'parsed_url': '/users/by-external/65.json'} Errors: ['The requested URL or resource could not be found.']

I'll try tweaking the API and see how it goes. I've noticed groups either don't or are really slow to update now. For example when someone joins corp or joins a group, it's a 50/50 shot of it making the update on Discourse.