[Bug]: no callback_manager field in OpenAIMultiModal?

[tomoqt]

Bug Description

I was palying around with multimodal indices, trying to get them to work with chat engines to read slides with multimodal support.

memory = ChatMemoryBuffer.from_defaults(token_limit=6000)

# Initialize your multimodal LLM and retrieval system
openai_mm_llm = OpenAIMultiModal(model="gpt-4-turbo",api_key='key', max_new_tokens=1500)
qa_tmpl = (

    "---------------------\n"
    "considerando la narrazione precedente ( a meno che non ce ne sia) narra la slide come se fossi un docente esperto. Non menzionare mai di essere un docente esperto, ma narrala come se di fronte a te ci fossero dei professionisti da formare "
    
    "slide: {query_str}\n"
 
)

storage_path = "C:/Users/epick/OneDrive/Desktop/Work/AI Sparks/Teknè Consulting/Tekne_audit_formazione/storage"
directory_path = "C:/Users/epick/OneDrive/Desktop/Work/AI Sparks/Teknè Consulting/Rag/tekne"
prompts = "Description for all documents" #dummy 
qa_template = PromptTemplate(qa_tmpl)
index = index_multimodal_documents_and_create_query_engines(directory_path, storage_path,prompts = prompts)
documents = SimpleDirectoryReader(storage_path).load_data()


chat_engine = index.as_chat_engine(chat_mode="openai", llm=openai_mm_llm,  memory=memory,
    image_qa_template=qa_template,
    similarity_top_k=3, 
    image_similarity_top_k=3,
    system_prompt=(
       "Sei un formatore specializzato in ambito automotive."
    ),
    )

but i get the following error:

    raise ValueError(f'"{self.__class__.__name__}" object has no field "{name}"')
ValueError: "OpenAIMultiModal" object has no field "callback_manager"

Version

0.10.32

Steps to Reproduce

build multimodal index, start a chat engine

Relevant Logs/Tracbacks

No response

[dosubot]

The error you're encountering with the OpenAIMultiModal class not recognizing a callback_manager field is due to the class not having this attribute defined or used. To address this issue, you'll need to modify the OpenAIMultiModal class to include handling for the callback_manager. Here's a concise guide on how to do it:

1. Add a Private Attribute for callback_manager: In the class definition of OpenAIMultiModal, introduce a private attribute to store the callback_manager.

   class OpenAIMultiModal(MultiModalLLM):
       _callback_manager: Optional[CallbackManager] = PrivateAttr()

2. Modify the __init__ Method: Update the constructor of the class to accept and store the callback_manager parameter.

   def __init__(
           self,
           model: str,
           temperature: float = DEFAULT_TEMPERATURE,
           max_new_tokens: Optional[int] = 300,
           callback_manager: Optional[CallbackManager] = None,
           **kwargs: Any,
       ) -> None:
       super().__init__(model=model, temperature=temperature, max_new_tokens=max_new_tokens, **kwargs)
       self._callback_manager = callback_manager

This adjustment ensures that the OpenAIMultiModal class can properly handle the callback_manager, storing it as a private attribute and making it available for use within the class methods as needed. If your implementation does not require a callback_manager, consider removing this parameter from the constructor to avoid the error.

Details

Sources

• llama-index-legacy/llama_index/legacy/multi_modal_llms/openai.py
• llama-index-core/llama_index/core/agent/react_multimodal/step.py
• llama-index-integrations/multi_modal_llms/llama-index-multi-modal-llms-openai/llama_index/multi_modal_llms/openai/base.py
• llama-index-core/llama_index/core/multi_modal_llms/base.py
• llama-index-core/llama_index/core/callbacks/base.py
• llama-index-legacy/llama_index/legacy/callbacks/base.py
• llama-index-core/llama_index/core/agent/custom/simple.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.