Ollama Support

[tomtom215]

As there is already support for llama.cpp, and that Langchain supports Ollama, it would be great to have the capability to use it with Langflow.

Thanks!

[alifzl]

dear langflow dev team. this feature could help a lot. kindly please consider adding this in feature releases.

[ogabrielluiz]

Hey

Downloading the model now. If it works easily we'll release it on 0.5.5 asap.

[alifzl]

that would be awsome. god bless.

[jeffhaskin]

That would be amazing.

[ogabrielluiz]

Hey, so.. while we don't have an official component you could try using this CustomComponent I wrote:

from langflow import CustomComponent

from langchain.llms.base import BaseLLM
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain.schema import Document
from langchain.llms import Ollama
from pathlib import Path
from typing import List, Optional

class Component(CustomComponent):
    display_name: str = "Ollama"
    description: str = "Run models served with Ollama"

    def build_config(self):
        config = {
            "mirostat": {
                "advanced": True,
            },
            "mirostat_eta": {
                "advanced": True,
            },
            "mirostat_tau": {
                "advanced": True,
            },
            "num_ctx": {
                "advanced": True,
            },
            "num_gpu": {
                "advanced": True,
            },
            "num_thread": {
                "advanced": True,
            },
            "repeat_last_n": {
                "advanced": True,
            },
            "repeat_penalty": {
                "advanced": True,
            },
            "temperature": {
                "advanced": False,
            },
            "stop": {
                "advanced": True,
            },
            "tfs_z": {
                "advanced": True,
            },
            "top_k": {
                "advanced": True,
            },
            "top_p": {
                "advanced": True,
            },
        }
        try:
            path = Path(
                "~/.ollama/models/manifests/registry.ollama.ai/library/"
            ).expanduser()
            # list all models
            models = [f for f in path.iterdir() if f.is_dir()]
            # get model names
            model_names = [f.name for f in models]
            model = {
                "required": True,
                "value": model_names[0],
                "options": model_names,
            }
        except Exception:
            model = {"required": True, "value": "llama2"}

        config["model"] = model
        return config

    def build(
        self,
        model: str,
        mirostat: Optional[int] = 0,
        mirostat_eta: Optional[float] = 0.1,
        mirostat_tau: Optional[float] = 5.0,
        num_ctx: Optional[int] = 2048,
        num_gpu: Optional[int] = None,
        num_thread: Optional[int] = None,
        repeat_last_n: Optional[int] = 64,
        repeat_penalty: Optional[float] = 1.1,
        temperature: Optional[float] = 0.8,
        stop: Optional[List[str]] = None,
        tfs_z: Optional[float] = 1.0,
        top_k: Optional[int] = 40,
        top_p: Optional[float] = 0.9,
    ) -> BaseLLM:
        llm = Ollama(
            model=model,
            mirostat=mirostat,
            mirostat_eta=mirostat_eta,
            mirostat_tau=mirostat_tau,
            num_ctx=num_ctx,
            num_gpu=num_gpu,
            num_thread=num_thread,
            repeat_last_n=repeat_last_n,
            repeat_penalty=repeat_penalty,
            temperature=temperature,
            stop=stop,
            tfs_z=tfs_z,
            top_k=top_k,
            top_p=top_p,
        )
        return llm

[alifzl]

so many thanks, lemmie check it in advance. I'd let you know about the results. cheers.

[alifzl]

dear @ogabrielluiz can you elaborate a little more about how can I use this Custom Component? I already have an Ollama docker container, running alongside with Langflow's docker-compose stack. my question is that how I can tell langflow to use my Ollama docker container and check it's containing models? considering the point that Ollama runtime is another docker container. should I share the .ollama directory of Ollama docker container with langflow's container so that it can access the model manifests and blobs?

[ogabrielluiz]

If you are running Langflow inside a container I'd say sharing the volume/folder might be the easiest way. Otherwise, you can just point to the model you want to use.

Regarding the CustomComponent, in the sidebar, find the Custom tab, drop the CustomComponent component on the screen and paste the code above in it, click "Check & Save" and you should be good to go.

[shayneoneill]

Has there been any progress on this? Ollama's rest api works just fine with langchain.

[vpb11]

nope, it does not ahaha

[vpb11]

2 days i have been trying so :/

[flefevre]

Dear all,

I have tried to connect LangFlow with Ollama server with a docker installation.
No way to connect them... (sorry for my english, i am french)

When i Put in langflow the different urls for ollama server, i get everytime the same error

• http://localhost:11434
• http://127.0.0.1:11434
• http://host.docker.internal:11434/

4usecaseOllamaOpenWeUILangflow.zip

here a zip with a basic configuration for ollama, openwebui and langflow
you just have to run

docker compose up

and to go http://localhost:7860/flow to import the ollama flow to test it.

I would appreciate your help.

Thanks.

angflow4      |                              │                                  │               
langflow4      |                              │ /home/user/.local/lib/python3.10 │               
langflow4      |                              │ /site-packages/starlette/websock │               
langflow4      |                              │ ets.py:111 in                    │               
langflow4      |                              │ _raise_on_disconnect             │               
langflow4      |                              │                                  │               
langflow4      |                              │   108 │                          │               
langflow4      |                              │   109 │   def _raise_on_disconne │               
langflow4      |                              │   110 │   │   if message["type"] │               
langflow4      |                              │ ❱ 111 │   │   │   raise WebSocke │               
langflow4      |                              │   112 │                          │               
langflow4      |                              │   113 │   async def receive_text │               
langflow4      |                              │   114 │   │   if self.applicatio │               
langflow4      |                              ╰──────────────────────────────────╯               
langflow4      |                              WebSocketDisconnect: (1005, None)                  
langflow4      | [2024-02-25 20:09:19 +0000] [46] [INFO] connection closed
langflow4      | [2024-02-25 20:09:19 +0000] [46] [INFO] ('172.27.0.1', 34476) - "WebSocket /api/v1/chat/8e4839cd-e8d2-4765-9a11-922692d8ae98?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIyY2RjYzAzYi03Mzk3LTRmNmEtYmY2YS01NmJhZDk2NWI2YzgiLCJleHAiOjE3NDA0MjcyMzB9.q23BWhTwU3uun82alujcc4J_o_XhTidbKmBhkfDJSe0" [accepted]
langflow4      | [2024-02-25 20:09:19 +0000] [46] [INFO] connection open
langflow4      | [02/25/24 20:09:25] ERROR    2024-02-25 20:09:25 - ERROR    -     service.py:216
langflow4      |                              service - Error handling websocket:                
langflow4      |                              (1005, None)                                       
langflow4      |                              Traceback (most recent call last):                 
langflow4      |                                                                                 
langflow4      |                                File                                             
langflow4      |                              "/home/user/.local/bin/langflow",                  
langflow4      |                              line 8, in <module>                                
langflow4      |                                  sys.exit(main())                               
langflow4      |                                  |   |    -> <function main at                  
langflow4      |                              0x7f1117adb7f0>                                    
langflow4      |                                  |   -> <built-in function exit>                
langflow4      |                                  -> <module 'sys' (built-in)>                   
langflow4      |                                File                                             
langflow4      |                              "/home/user/.local/lib/python3.10/si               
langflow4      |                              te-packages/langflow/__main__.py",                 
langflow4      |                              line 373, in main                                  
langflow4      |                                  app()                                          
langflow4      |                                  -> <typer.main.Typer object at                 
langflow4      |                              0x7f1117bf5ba0>                                    
langflow4      |                                File                                             
langflow4      |                              "/home/user/.local/lib/python3.10/si               
langflow4      |                              te-packages/typer/main.py", line                   
langflow4      |                              311, in __call__                                   
langflow4      |                                  return get_command(self)(*args,                
langflow4      |                              **kwargs)                                          
langflow4      |                                         |           |      |                    
langflow4      |                              -> {}                                              
langflow4      |                                         |           |      -> ()                
langflow4      |                                         |           ->                          
langflow4      |                              <typer.main.Typer object at                        
langflow4      |                              0x7f1117bf5ba0>                                    
langflow4      |                                         -> <function get_command                
langflow4      |                              at 0x7f1129663370>                                 
langflow4      |                                File                                             
langflow4      |                              "/home/user/.local/lib/python3.10/si               
langflow4      |                              te-packages/click/core.py", line                   
langflow4      |                              1157, in __call__                                  
langflow4      |                                  return self.main(*args,                        
langflow4      |                              **kwargs)                                          
langflow4      |                                         |    |     |       -> {}                
langflow4      |                                         |    |     -> ()                        
langflow4      |                                         |    -> <function                       
langflow4      |                              TyperGroup.main at 0x7f1129662ef0>                 
langflow4      |                                         -> <TyperGroup                          
langflow4      |                              main-entry-point>                                  
langflow4      |                                File                                             
langflow4      |                              "/home/user/.local/lib/python3.10/si               
langflow4      |                              te-packages/typer/core.py", line                   
langflow4      |                              778, in main                                       
langflow4      |                                  return _main(                                  
langflow4      |                                         -> <function _main at                   
langflow4      |                              0x7f1129662170>                                    
langflow4      |                                File                                             
langflow4      |                              "/home/user/.local/lib/python3.10/si               
langflow4      |                              te-packages/typer/core.py", line                   
langflow4      |                              216, in _main                 

[chrisnurse]

Quick suggestion trying to use the container name, perhaps

http://ollama:11434/

[orkutmuratyilmaz]

subscribing.