LibreChat just outputs nonsense text using LiteLLM and Ollama

[Linutux]

What happened?

When even asking a simple question, the answer is just nonsense:

You:
What's your name?

AI:
::ass:tieWhatassie User youristant your Corieass duass he Wi AssistantistantWhat' Assistant?ass: name:istantsistant User Assistant heWhatistant4:: he Cor:::tass Wi
' name Cor

That is my librechat.yaml:

version: 1.0.1
cache: true
# see more config in https://docs.librechat.ai/install/configuration/ai_endpoints.html
endpoints:
  custom:
    # LiteLLM
    - name: "XXX.de"
      apiKey: "sk-XXXXXXXXXXXX"
      baseURL: "https://api.xxx.xxx.de"
      models:
        default: ["openhermes:latest", "starling-lm:7b", "openchat:latest", "dolphin-mistral:latest", "mistral-openorca:latest", "stable-code:latest"]
        fetch: true
      titleConvo: true
      titleModel: "openhermes:latest"
      summarize: false
      summaryModel: "openhermes:latest"
      forcePrompt: false
      modelDisplayLabel: "AI"
      drop_params: true
      stream: true

Steps to Reproduce

1. Install LibreChat in docker
2. Connect to LiteLLM in docker
3. Connect LiteLLM to Ollama in docker

What browsers are you seeing the problem on?

Firefox, Mobile (Android)

Relevant log output

No response

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

You can try adjusting the temperature according to what's recommended for the model you are using. That might be affecting the response.

    addParams:
        - temperature: 0.5

https://docs.librechat.ai/install/configuration/custom_config.html#addparams

[Linutux]

Unfortunately, that didn't help.

It's on every model, not just one.

When connecting to LiteLLM via Chatbox AI (and other UIs) everything works fine. There must be some sort of difference in configuration.

[danny-avila]

It's probably one of the default parameters that is affecting every model: https://docs.librechat.ai/install/configuration/custom_config.html#default-parameters

You can try removing them all with dropParams which I noticed you have set to drop_params. This is not configured right:

drop_params: true # won't do anything

The correct configuration for this field is shown here as dropParams: https://docs.librechat.ai/install/configuration/custom_config.html#dropparams

You should specify which params to drop.

Tagging @berry-13 and @bsu3338 who use Ollama with LibreChat and have not reported this issue.

[danny-avila]

FWIW I just tested Ollama with mistral:latest with no issues and the following config:

    - name: "Ollama"
      apiKey: "ollama"
      baseURL: "http://localhost:11434/v1/"
      models:
        default: [
          "mistral",
          ]
        fetch: false # fetching list of models is not supported
      titleConvo: true
      titleModel: "mistral"
      summarize: false
      summaryModel: "mistral"
      forcePrompt: false
      modelDisplayLabel: "Ollama"

I have the latest version of Ollama and I ran ollama serve and setup my config as above

[berry-13]

I have experienced no issues with any of the models. Everything seems to be working fine for me as well

[bsu3338]

What is your version of LibreChat, Litellm, and Ollama?

[Linutux]

I have Ollama BEHIND LiteLLM. So I connect from LibreChat to LiteLLM, which then connects to Ollama.

When Using other UIs, everything works fine.

The latest Answer I got was

<|im_start|>assttassistantistantie he heißtassistantuser?t

The "im_start" seems to be some sort of "start of message" signal of the API. But it seems to not be interpreted right.

[Linutux]

This is what I get in the logs:

2024-03-04 21:31:53 info: Server listening on all interfaces at port 5000. Use http://localhost:5000 to access it
2024-03-04 21:33:02 error: [MeiliMongooseModel.findOneAndUpdate] Convo not found in MeiliSearch and will index 991019cc-c59c-4b70-8229-88ba127e7dfe Document 991019cc-c59c-4b70-8229-88ba127e7dfe not found.
2024-03-04 21:33:09 warn: [OpenAIClient.chatCompletion][create] API error
2024-03-04 21:33:09 error: [OpenAIClient.chatCompletion] Unhandled error type Error: 500 cannot pickle 'async_generator' object

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/litellm/proxy/proxy_server.py", line 2714, in chat_completion
response = await proxy_logging_obj.post_call_success_hook(
File "/usr/local/lib/python3.9/site-packages/litellm/proxy/utils.py", line 394, in post_call_success_hook
new_response = copy.deepcopy(response)
File "/usr/local/lib/python3.9/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle 'async_generator' object

2024-03-04 21:33:09 error: [OpenAIClient] There was an issue generating the title with the completion method Error: 500 cannot pickle 'async_generator' object

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/litellm/proxy/proxy_server.py", line 2714, in chat_completion
response = await proxy_logging_obj.post_call_success_hook(
File "/usr/local/lib/python3.9/site-packages/litellm/proxy/utils.py", line 394, in post_call_success_hook
new_response = copy.deepcopy(response)
File "/usr/local/lib/python3.9/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle 'async_generator' object

[shuther]

I found the problem (for me). While reading the LibreChat/logs/debug*.log I discovered that the default end point was gptPlugins instead of ollama/litellm.

[shuther]

sorry, direct Ollama is working, but I still face issues with litellm :( I opened another issue