use open source models like LLaMa instead of openAI APIs.

[Fadlo31]

i don't fully grasp this subject about models and AIs ... but i think we can use free and open models that run on our machines and prompt them .

[jakderrida]

I think the problem most often faced is that, while those using LLaMa's might think the rest of us are wasting money using the OpenAI API, using the LLaMa isn't really free and, for anything comparable to GPT-4's results, requires NVLinking like 4-8 RTX GPUs.My point is that calling it "free", when it comes to running it locally, is very deceptive. Also, while I wish it weren't so, my analysis of energy costs to fine tune, train, or inference one of the most robust models I could find data on (I think it was like a 65B parameter LLaMa derivative) demonstrated that the electric costs alone exceeded the equivalent, per prompt, API costs. So even after I buy all those massive GPUs, I'm paying out more than OpenAI will charge me. I don't wanna discourage the evolution of LLaMa or Open-source (which I'm excited about), but everyone calling these things free must be teenagers that only view it as free because they weren't the ones that paid for their $3700 gaming rig, nor do they conceive that all the megawatts it draws are not really free.

EDIT: Let me qualify my assessment by saying that I believe that, if I understand LoRa correctly, there are massive exceptions to what I've said. Should you use GPT-4 questions and answers to create a specialized LLaMa such that it addresses very frequent questions with a very narrow skillset, that would definitely reduce power needed.
Basically, training a LLaMa with LoRa and a problem set that focuses exclusively on spotting GPT-4's failure to make sure the entire response is in JSON form, along with specializing in rewriting it appropriately, that would no doubt be very efficient.
Basically, it would be like having an employee that lacks education focus on spotting defects along the assembly line and knowing exactly how to deal with them. No reason to ask the CEO to step in.

[Fadlo31]

I totally understand and recognize the misuse of the word "free." What I meant is not using the 65B model, which is very expensive to run. Instead, I suggest using the 7B or 13B model and utilizing a "tool" like "superAGI" to prompt these "lightweight models" multiple times in order to correct and fix "artificial hallucinations." Please note that I don't fully grasp the subject.

In recent news, a model named Orca with 13B parameters is going to be open source and is expected to have very similar quality to GPT-3.5 and GPT-4. You can find the paper at: https://arxiv.org/pdf/2306.02707.pdf

I don't think it's easy (as I have only grasped the subject with difficulty), but having the option to choose between an owned model and a paid API call to models is a game changer. My thought process is to use this tool to create multiple instances or prompts, then compare the output and select the best answer.

[chris-aeviator]

Llama 2 70B is runable on a single A6000

[neelayan7]

We have already implemented Local LLMs in this PR: #289
Closing this.

[neelayan7]

If you have any other requirements, feel free to open this discussion!