Talking to 3rd Party LLMs similar to LocalAiChatModel.java?

[schadha101]

We tried using LocalAiChatModel.java to connect to a LLaMA model hosted on runpod. However we get error 404.

Questions.

1. Is LocalAiChatModel.java only for OpenAI or can it be used be used to talk to Opensource models?
2. Does the model have to be installed localhost or it can be hosted on a public server. This I think is any public server.

If above is not available, we will build: HostedAiChatModel.java to chat with models accessible through REST API.

Looking forward to the response.

[langchain4j]

Hi @schadha101 LocalAiChatModel is supposed to work with LLMs hosted via LocalAI or oobabooga's OpenAI API, you can run Llama and other open models with them. It can be installed anywhere, including localhost. You have to provide a valid baseUrl for it to work properly. Langchain4j will append /completions, /chat/completions, /embeddings, etc to the baseUrl depending on the LocalAi*Model class used.

[schadha101]

Thanks. We used this code and it gives us error 404

ChatLanguageModel model = LocalAiChatModel.builder()
                .baseUrl("https://k9dxl4sl7fqgsg-80.proxy.runpod.net")
                .modelName("meta-llama/Llama-2-13b-chat-hf")
                .maxTokens(3)
                .logRequests(true)
                .logResponses(true)
                .timeout(Duration.ofMinutes(1))
                .build();

        String answer = model.generate("Say 'hello'");


         I reviewed the localAI I website and it seems to download the LLM which I do not think is a good idea when we work with large LL M's

We can access that endpoints using python

[langchain4j]

Which tool are you using to run LLM on runpod? Regarding the downloading of LLMs, what do you mean? If you want to run LLMs locally you will need to download them first.

[schadha101]

Which tool are you using to run LLM on runpod?

Abdul si CCed to this email says he run python code. Abdul provide code below Should we follow these steps on runpod https://localai.io/basics/getting_started/? Code to setup LLM on Run Pod import requests import runpod from text_generation import Client runpod.api_key='UXJMA28ZXX6U3GSFVLX57IPF1RWPXWDTCKIV656X' num_shard = 1 quantize = "bitsandbytes" pod = runpod.create_pod( name="Llama-13b-chat", image_name="ghcr.io/huggingface/text-generation-inference:1.0.3", gpu_type_id="NVIDIA RTX 6000 Ada Generation", cloud_type="COMMUNITY", docker_args=f"--model-id meta-llama/Llama-2-13b-chat-hf --num-shard {num_shard} --quantize {quantize} --max-input-length 4095 --max-total-tokens 4096", gpu_count=num_shard, volume_in_gb=100, container_disk_in_gb=100, ports="80/http", volume_mount_path="/data", env={'HUGGING_FACE_HUB_TOKEN':'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'} ) print(pod)

Regarding the downloading of LLMs, what do you mean?

https://localai.io/model-compatibility/ LocalAI will attempt to automatically load models which are not explicitly configured for a specific backend. You can specify the backend to use by configuring a model with a YAML file. See <https://localai.io/advanced/> the advanced section for more details.

If you want to run LLMs locally you will need to download them first.

We do not want to run the LLMs locally. With best regards, Sanjay Chadha <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fsanjaychadha101%2F&data=04%7C01%7CCSauls%40libertyhomecare.com%7Cbca7d008b61d43be6b5608d94ba202f6%7C9ccd3796bf604eb48f536b78ff39be60%7C0%7C0%7C637623981483973788%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nR%2FvtIYKhkn2eD0KOn%2FD2sJqrTW4%2Fn9JRi0eVoqSJIU%3D&reserved=0> Phone: 613-203-6162 Toll-free: 844-800-7233 ***@***.***> ***@***.*** <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sirona.tv%2F&data=04%7C01%7CCSauls%40libertyhomecare.com%7Cbca7d008b61d43be6b5608d94ba202f6%7C9ccd3796bf604eb48f536b78ff39be60%7C0%7C0%7C637623981483983787%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=WOuaLbAo%2FxH%2B6AqUt7tKOXDpcyu7HrkmrbNXKjB2ccY%3D&reserved=0> www.Sirona.tv <http://www.aionscheduler.com/> http://www.AIONScheduler.com This email is intended for the addressee(s) & may contain confidential information. From: LangChain4j ***@***.*** Sent: October 20, 2023 12:36 PM To: langchain4j/langchain4j Cc: Sanjay Chadha; Mention Subject: Re: [langchain4j/langchain4j] Talking to 3rd Party LLMs similar to LocalAiChatModel.java? (Discussion #243) Which tool are you using to run LLM on runpod? Regarding the downloading of LLMs, what do you mean? If you want to run LLMs locally you will need to download them first. — Reply to this email directly, view it on GitHub <#243 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHXGYT7XJPP7GFWCLZ42CVTYAKSANAVCNFSM6AAAAAA6JBBMUKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNBRGE4TC> . You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AHXGYT3LL7KAZRFHQO44TDLYAKSANA5CNFSM6AAAAAA6JBBMUKWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAOACIO.gif> Message ID: ***@***.***>

[langchain4j]

@schadha101 you might want to remove your API keys from your latest message. If you do not want to run LLMs locally, what are you trying to achieve? From your code it seems that you are trying to run Llama2 on runpod, which is considered "running locally" (as opposed to using some third party service like OpenAI)

[schadha101]

If running Llama2 on runpod is considered running locally then yes we want to run locally! I will remove the API keys. Thanks. I found these instructions to install “LocalAI” on runpod. https://github.com/fHachenberg/local-ai-ootb If you have experience with this, pls share. With best regards, Sanjay Chadha <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fsanjaychadha101%2F&data=04%7C01%7CCSauls%40libertyhomecare.com%7Cbca7d008b61d43be6b5608d94ba202f6%7C9ccd3796bf604eb48f536b78ff39be60%7C0%7C0%7C637623981483973788%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nR%2FvtIYKhkn2eD0KOn%2FD2sJqrTW4%2Fn9JRi0eVoqSJIU%3D&reserved=0> Phone: 613-203-6162 Toll-free: 844-800-7233 ***@***.***> ***@***.*** <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sirona.tv%2F&data=04%7C01%7CCSauls%40libertyhomecare.com%7Cbca7d008b61d43be6b5608d94ba202f6%7C9ccd3796bf604eb48f536b78ff39be60%7C0%7C0%7C637623981483983787%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=WOuaLbAo%2FxH%2B6AqUt7tKOXDpcyu7HrkmrbNXKjB2ccY%3D&reserved=0> www.Sirona.tv <http://www.aionscheduler.com/> http://www.AIONScheduler.com This email is intended for the addressee(s) & may contain confidential information. From: LangChain4j ***@***.*** Sent: October 20, 2023 2:29 PM To: langchain4j/langchain4j Cc: Sanjay Chadha; Mention Subject: Re: [langchain4j/langchain4j] Talking to 3rd Party LLMs similar to LocalAiChatModel.java? (Discussion #243) @schadha101 <https://github.com/schadha101> you might want to remove your API keys from your latest message. If you do not want to run LLMs locally, what are you trying to achieve? From your code it seems that you are trying to run Llama2 on runpod, which is considered "running locally" (as opposed to using some third party service like OpenAI) — Reply to this email directly, view it on GitHub <#243 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHXGYTZGDSEV5PYEZCDZQFLYAK7IHAVCNFSM6AAAAAA6JBBMUKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNBSGI2TS> . You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AHXGYTZ2W7JMM7WBVQMFT2LYAK7IHA5CNFSM6AAAAAA6JBBMUKWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAOAELG.gif> Message ID: ***@***.***>