Agent exceeded context length while iterating to try and self-fix an error in a tool

[shhlife]

• [X ] I have read and agree to the contributing guidelines.

Describe the bug
I have a custom tool that interacts with ShotGrid. It's given to an Agent. The agent was performing an action that requires a certain format, and it kept getting an error from the ShotGrid api. The agent was using COT to try and fix the prompt, but eventually got an error that it exceeded the model's maximum context length.

Additional context
Here's the output:

Thought: The user is asking about the process of creating the complex filter and retrieving the data. I need to use the ShotgridClient action to retrieve the entities that match the complex filter. The filter should include conditions for the creation date to be in November 2023, the entity type to be "Character", and the name not to contain the letter 'o'.
Action: 
```json
{
  "name": "ShotgridClient",
  "path": "retrieve_matching_entities",
  "input": {
    "values": {
      "entity_type": "Character",
      "filters": [
        ["created_at", "between", ["2023-11-01", "2023-11-30"]],
        ["name", "not_contains", "o"]
      ],
      "fields": ["id", "name", "created_at"]
    }
  }
}

Thought: The error message indicates that "Character" is not a valid entity type. I need to check the valid entity types and find the correct one for characters. The VectorStoreClient action can be used to find the correct entity type for characters in ShotGrid.
Action: {"name": "VectorStoreClient", "path": "search", "input": {"values": {"query": "What is the entity type for characters in ShotGrid?"}}}Thought: The VectorStoreClient action was unable to find the entity type for characters in ShotGrid. However, based on the list of valid entity types provided in the error message, the entity type for characters is likely "Asset". I will use this entity type in the ShotgridClient action to retrieve the entities that match the complex filter.

Action: {"name": "ShotgridClient", "path": "retrieve_matching_entities", "input": {"values": {"entity_type": "Asset", "filters": [["created_at", "between", ["2023-11-01", "2023-11-30"]], ["name", "not_contains", "o"]], "fields": ["id", "name", "created_at"]}}}

Thought: The error message indicates that the date format is incorrect. I need to use the DateTime action to get the correct date format for November 2023.

Action: {"name": "DateTime", "path": "get_relative_datetime", "input": {"values": {"relative_date_string": "2023-11-01"}}}

Thought: The DateTime action returned the correct date format for November 1, 2023. I need to do the same for November 30, 2023.

Action: {"name": "DateTime", "path": "get_relative_datetime", "input": {"values": {"relative_date_string": "2023-11-30"}}}

Thought: Now that I have the correct date format for both the start and end of November 2023, I can use these dates in the ShotgridClient action to retrieve the entities that match the complex filter.

Action: {"name": "ShotgridClient", "path": "retrieve_matching_entities", "input": {"values": {"entity_type": "Asset", "filters": [["created_at", "between", ["2023-11-01 00:00:00", "2023-11-30 00:00:00"]], ["name", "not_contains", "o"]], "fields": ["id", "name", "created_at"]}}}

Thought: The error message indicates that the date format is still incorrect. The correct format should include the time in UTC or with a UTC offset. I need to adjust the date format to include the time in UTC.

Action: {"name": "DateTime", "path": "get_relative_datetime", "input": {"values": {"relative_date_string": "2023-11-01T00:00:00Z"}}}

Thought: The DateTime action returned the correct date format for November 1, 2023, with the time in UTC. I need to do the same for November 30, 2023.

Action: {"name": "DateTime", "path": "get_relative_datetime", "input": {"values": {"relative_date_string": "2023-11-30T00:00:00Z"}}}

Thought: Now that I have the correct date format for both the start and end of November 2023, I can use these dates in the ShotgridClient action to retrieve the entities that match the complex filter.

Action: {"name": "ShotgridClient", "path": "retrieve_matching_entities", "input": {"values": {"entity_type": "Asset", "filters": [["created_at", "between", ["2023-11-01T00:00:00Z", "2023-11-30T00:00:00Z"]], ["name", "not_contains", "o"]], "fields": ["id", "name", "created_at"]}}}

Thought: The error message indicates that the "name" field does not exist for the "Asset" entity type. I need to use the VectorStoreClient action to find the correct field for the name of assets in ShotGrid.

Action: {"name": "VectorStoreClient", "path": "search", "input": {"values": {"query": "What is the field for the name of assets in ShotGrid?"}}}

Thought: The VectorStoreClient action returned that the field for the name of assets in ShotGrid is 'code'. I can use this field in the ShotgridClient action to retrieve the entities that match the complex filter.

Action: {"name": "ShotgridClient", "path": "retrieve_matching_entities", "input": {"values": {"entity_type": "Asset", "filters": [["created_at", "between", ["2023-11-01T00:00:00Z", "2023-11-30T00:00:00Z"]], ["code", "not_contains", "o"]], "fields": ["id", "code", "created_at"]}}}[12/01/23 06:15:45] 

ERROR    ToolkitTask 3bff3626eba7498d836a0e50f1e13e77                                                                           
                             Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens. However, your messages    
                             resulted in 8694 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param':         
                             'messages', 'code': 'context_length_exceeded'}}                                                                        
                             Traceback (most recent call last):                                                                                     
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/gripta
                             pe/tasks/base_task.py", line 86, in execute                                                                            
                                 self.output = self.run()                                                                                           
                                               ^^^^^^^^^^                                                                                           
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/gripta
                             pe/tasks/toolkit_task.py", line 133, in run                                                                            
                                 ActionSubtask(self.active_driver().run(prompt_stack=self.prompt_stack).to_text())                                  
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                             
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/gripta
                             pe/drivers/prompt/base_prompt_driver.py", line 67, in run                                                              
                                 for attempt in self.retrying():                                                                                    
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/tenaci
                             ty/__init__.py", line 347, in __iter__                                                                                 
                                 do = self.iter(retry_state=retry_state)                                                                            
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                            
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/tenaci
                             ty/__init__.py", line 314, in iter                                                                                     
                                 return fut.result()                                                                                                
                                        ^^^^^^^^^^^^                                                                                                
                               File                                                                                                                 
                             "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/
                             _base.py", line 449, in result                                                                                         
                                 return self.__get_result()                                                                                         
                                        ^^^^^^^^^^^^^^^^^^^                                                                                         
                               File                                                                                                                 
                             "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/
                             _base.py", line 401, in __get_result                                                                                   
                                 raise self._exception                                                                                              
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/gripta
                             pe/drivers/prompt/base_prompt_driver.py", line 74, in run                                                              
                                 for chunk in completion_chunks:                                                                                    
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/gripta
                             pe/drivers/prompt/openai_chat_prompt_driver.py", line 84, in try_stream                                                
                                 result = self.client.chat.completions.create(**self._base_params(prompt_stack), stream=True)                       
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                       
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/openai
                             /_utils/_utils.py", line 299, in wrapper                                                                               
                                 return func(*args, **kwargs)                                                                                       
                                        ^^^^^^^^^^^^^^^^^^^^^                                                                                       
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/openai
                             /resources/chat/completions.py", line 598, in create                                                                   
                                 return self._post(                                                                                                 
                                        ^^^^^^^^^^^                                                                                                 
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/openai
                             /_base_client.py", line 1063, in post                                                                                  
                                 return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))                          
                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                           
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/openai
                             /_base_client.py", line 842, in request                                                                                
                                 return self._request(                                                                                              
                                        ^^^^^^^^^^^^^^                                                                                              
                               File                                                                                                                 
                             "/Users/jason/Library/Caches/pypoetry/virtualenvs/chatbot-streaming-ubAWX4gC-py3.11/lib/python3.11/site-packages/openai
                             /_base_client.py", line 885, in _request                                                                               
                                 raise self._make_status_error_from_response(err.response) from None                                                
                             openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens.   
                             However, your messages resulted in 8694 tokens. Please reduce the length of the messages.", 'type':                    
                             'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}                                      

[derekpai]

Jason was able to reproduce:

I got a different error but I think it's related..
I made an agent run a tool with an intentional mistake in their command so they would get errors.
I asked them to do it 20 times and to force themselves to get it wrong, because I think that's what caused the context length issue.
After 18 times, I got this message:

[2024-03-18 23:10:14,849: ERROR/ForkPoolWorker-1] Error handling event: The request to the Slack API failed. (url: https://www.slack.com/api/chat.update)
worker-1    | The server responded with: {'ok': False, 'error': 'no_text'}
worker-1    | [2024-03-18 23:10:14,853: ERROR/ForkPoolWorker-1] Traceback (most recent call last):
worker-1    |   File "/app/griptape_slackbot/handlers/base_handler.py", line 66, in handle
worker-1    |     self.try_handle(args)
worker-1    |   File "/app/griptape_slackbot/handlers/events/message_event_handler.py", line 21, in try_handle
worker-1    |     self._process_event(args)
worker-1    |   File "/app/griptape_slackbot/handlers/events/base_event_handler.py", line 80, in _process_event
worker-1    |     self._run_agent(
worker-1    |   File "/app/griptape_slackbot/handlers/base_handler.py", line 443, in _run_agent
worker-1    |     post_update(final_blocks)
worker-1    |   File "/app/griptape_slackbot/handlers/base_handler.py", line 223, in post_update
worker-1    |     slack_client.chat_update(
worker-1    |   File "/app/.venv/lib/python3.11/site-packages/slack_sdk/web/client.py", line 2686, in chat_update
worker-1    |     return self.api_call("chat.update", json=kwargs)
worker-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1    |   File "/app/.venv/lib/python3.11/site-packages/slack_sdk/web/base_client.py", line 155, in api_call
worker-1    |     return self._sync_send(api_url=api_url, req_args=req_args)
worker-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1    |   File "/app/.venv/lib/python3.11/site-packages/slack_sdk/web/base_client.py", line 186, in _sync_send
worker-1    |     return self._urllib_api_call(
worker-1    |            ^^^^^^^^^^^^^^^^^^^^^^
worker-1    |   File "/app/.venv/lib/python3.11/site-packages/slack_sdk/web/base_client.py", line 317, in _urllib_api_call
worker-1    |     ).validate()
worker-1    |       ^^^^^^^^^^
worker-1    |   File "/app/.venv/lib/python3.11/site-packages/slack_sdk/web/slack_response.py", line 199, in validate
worker-1    |     raise e.SlackApiError(message=msg, response=self)
worker-1    | slack_sdk.errors.SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/chat.update)
worker-1    | The server responded with: {'ok': False, 'error': 'no_text'}
worker-1    |