[Bug]: pydantic.v1.error_wrappers.ValidationError: 1 validation error for Generation type unexpected value; permitted: 'Generation' (type=value_error.const; given=ChatGeneration; permitted=('Generation',))

[sdd031215]

Current Behavior

def init_gptcache_map(cache_obj: Cache):
cache_base = CacheBase('sqlite')
vector_base = VectorBase('milvus', host='xx', port='19530', dimension=len(cache_embeddings.to_embeddings('abc')), collection_name='chatbot')
data_manager = get_data_manager(cache_base, vector_base)
cache_obj.init(
pre_embedding_func=get_content_func,
embedding_func=cache_embeddings.to_embeddings,
data_manager=data_manager,
similarity_evaluation=SearchDistanceEvaluation(), #ExactMatchEvaluation()
)
langchain.llm_cache = GPTCache(init_gptcache_map)

The following error occurred during the execution of langchain's tool:
" pydantic.v1.error_wrappers.ValidationError: 1 validation error for Generation type unexpected value; permitted: 'Generation' (type=value_error.const; given=ChatGeneration; permitted=('Generation',))"

@SimFG

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

[aravindarc]

Any update on resolving this issue or any workaround, if someone guides me I want to attempt fixing this

[SimFG]

@aravindarc If you also have this problem, you can give me the error stack, and then I can look at the cause of the problem and give you a certain solution.

[Aneerudh2k2]

Hey guys, same issue here code remains same as @sdd031215, and here is the error stack below, can you help me @SimFG
@aravindarc @technicalpickles @jmahmood ?

Traceback (most recent call last):
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/flask/app.py", line 1488, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/flask/app.py", line 1466, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
                                                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/Documents/my documents/project_files/folder_name/nodejs-backend/src/langchain.microservice/app.py", line 754, in summary_gen
    response = ask_query_to_qa_chain(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/Documents/my documents/project_files/folder_name/nodejs-backend/src/langchain.microservice/app.py", line 421, in ask_query_to_qa_chain
    qa_chain(
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_core/_api/deprecation.py", line 145, in warning_emitting_wrapper
    return wrapped(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/base.py", line 378, in __call__
    return self.invoke(
           ^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/base.py", line 163, in invoke
    raise e
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/base.py", line 153, in invoke
    self._call(inputs, run_manager=run_manager)
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/combine_documents/base.py", line 137, in _call
    output, extra_return_dict = self.combine_docs(
                                ^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/combine_documents/stuff.py", line 244, in combine_docs
    return self.llm_chain.predict(callbacks=callbacks, **inputs), {}
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/llm.py", line 293, in predict
    return self(kwargs, callbacks=callbacks)[self.output_key]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_core/_api/deprecation.py", line 145, in warning_emitting_wrapper
    return wrapped(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/base.py", line 378, in __call__
    return self.invoke(
           ^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/base.py", line 163, in invoke
    raise e
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/base.py", line 153, in invoke
    self._call(inputs, run_manager=run_manager)
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/llm.py", line 103, in _call
    response = self.generate([inputs], run_manager=run_manager)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain/chains/llm.py", line 115, in generate
    return self.llm.generate_prompt(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 544, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 408, in generate
    raise e
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 398, in generate
    self._generate_with_cache(
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 585, in _generate_with_cache
    cache_val = llm_cache.lookup(prompt, llm_string)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_community/cache.py", line 807, in lookup
    return [
           ^
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_community/cache.py", line 808, in <listcomp>
    Generation(**generation_dict) for generation_dict in json.loads(res)
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/langchain_core/load/serializable.py", line 120, in __init__
    super().__init__(**kwargs)
  File "/Users/aneerudhm/anaconda3/envs/llm/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Generation
type
  unexpected value; permitted: 'Generation' (type=value_error.const; given=ChatGeneration; permitted=('Generation',))

[SimFG]

@Aneerudh2k2 From the error stack, I don’t seem to see anything related to gptcache. Can you try other caches to see if there are similar issues?

[Aneerudh2k2]

@SimFG this error persists after using gptcache, i tried different cache like redis error didn't occur. I guess the issue from langchain integration with gptcache, because the common thing is with me and @sdd031215 is langchain. Let me try wit different approach if works let me post it over here.

btw thanks : )

[SimFG]

@aravindarc thanks your feedback. and I will confirm this asap

[oussamaJmaaa]

any update regarding this issue ?

[SimFG]

@oussamaJmaaa I am confirming step by step and will reply as soon as the result is available

[SimFG]

Hi, @oussamaJmaaa @Aneerudh2k2 I use the latest langchain and gptcache lib, and there seem be no error.
You can use the latest version to check it, and you meet other errors, please tell me the error stack and the demo code to expedite problem resolution.

Here is my code:

import hashlib
import timeit

from gptcache import Cache
from gptcache.adapter.api import init_similar_cache
from gptcache.processor.pre import get_prompt
from gptcache.manager.factory import manager_factory
from langchain.globals import set_llm_cache
from langchain_community.cache import GPTCache

from langchain.globals import set_llm_cache
from langchain_openai import OpenAI

# def init_gptcache(cache_obj: Cache, llm: str):
#     cache_obj.init(
#         pre_embedding_func=get_prompt,
#         data_manager=manager_factory(
#             manager="map",
#             data_dir=f"map_cache_{llm}"
#         ),
#     )

def get_hashed_name(name):
    return hashlib.sha256(name.encode()).hexdigest()


def init_gptcache(cache_obj: Cache, llm: str):
    hashed_llm = get_hashed_name(llm)
    init_similar_cache(cache_obj=cache_obj, data_dir=f"similar_cache_{hashed_llm}")

set_llm_cache(GPTCache(init_gptcache))

llm = OpenAI(model_name="gpt-3.5-turbo-instruct", n=2, best_of=2)

execution_time = timeit.timeit(lambda: llm("Tell me a joke"), number=1)
print(f"Execution time: {execution_time} seconds")

execution_time = timeit.timeit(lambda: llm("Tell me a joke"), number=1)
print(f"Execution time: {execution_time} seconds")

execution_time = timeit.timeit(lambda: llm("Tell me joke"), number=1)
print(f"Execution time: {execution_time} seconds")

the test result:

the env and lib version:

Python 3.10.13

Name: langchain
Version: 0.1.9

Name: gptcache
Version: 0.1.43

[oussamaJmaaa]

thank you for your help , your code works fine but when i changed the llm instance to llm = ChatOpenAI(model_name=llm_name, temperature=0) and used a ConversationalRetrieval chain , it gives the same error again after the second execution , can you please help me with that ?
Here's the code :

def get_hashed_name(name):
return hashlib.sha256(name.encode()).hexdigest()

def init_gptcache(cache_obj: Cache, llm: str):
hashed_llm = get_hashed_name(llm)
init_similar_cache(cache_obj=cache_obj, data_dir=f"cache2{hashed_llm}")
set_llm_cache(GPTCache(init_gptcache))

llm = ChatOpenAI(model_name="gpt-3.5-turbo")

embedding = OpenAIEmbeddings()
vector_store = Chroma(persist_directory="my_directory", embedding_function=embedding)

template = """
You are a customer service bot. Respond to the user's input in the
same language they used. Use the given context to provide a brief yet
informative response. If unsure, just say so. Finish by saying
"Thanks for asking!"\n \n\n{context}\n*\nQuestion: {question}\nHelpful Answer:"""
QA_CHAIN_PROMPT = PromptTemplate.from_template(template)

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
retriever = vector_store.as_retriever(k=5)

qa = ConversationalRetrievalChain.from_llm(
llm=llm,
retriever=retriever,
memory=memory,
combine_docs_chain_kwargs={"prompt": QA_CHAIN_PROMPT},
verbose = False
)
user_input =" hello !"
response = qa({"question": user_input})
print(response["answer"])

[SimFG]

@oussamaJmaaa thanks you share your demo code, i will check it tomorrow.
But judging from the current results, I guess there is a high probability that it is not a problem with GPTCache, because GPTCache is implemented based on the cache interface like other caches.

[oussamaJmaaa]

hi , thank you for your reply , the code works fine without GPTCache and when i used it to cache the results , the first execution of the code works fine and saves data into to the sqlite db but after executing a second i get that error ..

[SimFG]

@oussamaJmaaa
I carefully read the cache code in the langchain part, and I found that the problem was not caused by gptcache, but that there was a problem with langchain's cache processing.

Code to store results in GPTCache:

    def update(self, prompt: str, llm_string: str, return_val: RETURN_VAL_TYPE) -> None:
        """Update cache.
        First, retrieve the corresponding cache object using the `llm_string` parameter,
        and then store the `prompt` and `return_val` in the cache object.
        """
        for gen in return_val:
            if not isinstance(gen, Generation):
                raise ValueError(
                    "GPTCache only supports caching of normal LLM generations, "
                    f"got {type(gen)}"
                )
        from gptcache.adapter.api import put

        _gptcache = self._get_gptcache(llm_string)
        handled_data = json.dumps([generation.dict() for generation in return_val])
        put(prompt, handled_data, cache_obj=_gptcache)
        return None

Code to get results from GPTCache:

    def lookup(self, prompt: str, llm_string: str) -> Optional[RETURN_VAL_TYPE]:
        """Look up the cache data.
        First, retrieve the corresponding cache object using the `llm_string` parameter,
        and then retrieve the data from the cache based on the `prompt`.
        """
        from gptcache.adapter.api import get

        _gptcache = self._get_gptcache(llm_string)

        res = get(prompt, cache_obj=_gptcache)
        if res:
            return [
                Generation(**generation_dict) for generation_dict in json.loads(res)
            ]
        return None

According to the error stack, there should be an error during json.loads. That is, the format of the saved result and the parsing format should be inconsistent, which caused this error. I guess this may be related to the result processing of the conversation type llm, because if it is just openai, there will be no such error. However, the current langchain is too complicated and is no longer what I originally understood it to be. I can no longer fix this error.

[oussamaJmaaa]

i see , thank you for your reply , is there any alternative ways to use langchain RetrievalChain with memory and gptcache ?

[SimFG]

@oussamaJmaaa i have no idea about this, and i think it need to be fixed by the langchain community

[oussamaJmaaa]

OKay , thanks a lot !

[theinhumaneme]

Hello I have looked into the code and when changing Generation to ChatGeneration in the cache.py the cache seems to work fine and is responsive back again, it seems like langchain has moved to ChatGeneration from Generation in the internal working and the cache.py could have fallen behind.

I don't know if this is the right approach to solve this problem. but it works :D (locally)

@SimFG @oussamaJmaaa can I make pull request in this regard for this change?

[SimFG]

@theinhumaneme of course, do it!