Where can I learn about the different models, types of models and fine-tuning?

[dportabella]

I have some books in PDF and I want to query the books using localGPT.

I have a machine with proxmox (server virtualization). Inside proxmox, I have a VM with 24 GB mem, 20 processors (1 socket, 20 cores), and NVDIA GeForce RTX 4070 with 16 GB VRAM (directly attached from host to the VM). Is this a good setup for optimal and fast replies?

Which is the best powerful model available for this task?

Where can I learn about the different models and fine-tuning?
"TheBloke/Llama-2-7B-Chat-GGML"
"TheBloke/vicuna-7B-1.1-HF"
"TheBloke/Wizard-Vicuna-7B-Uncensored-HF"
"TheBloke/guanaco-7B-HF"
'NousResearch/Nous-Hermes-13b'

I see that GPTQ models are for small device such as phones, so I don't need those.
Should I use HF models or GGML (quantized cpu+gpu+mps) models?

[dportabella]

I have NVDIA GeForce RTX 4070 with 12GB. The best models I can use are the 7B GPTQ 8bit models, right?
What is the difference between the Vicuna and the LM models?

### 7b GPTQ Models for 8GB GPUs
# MODEL_ID = "TheBloke/Wizard-Vicuna-7B-Uncensored-GPTQ"
# MODEL_BASENAME = "Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act.order.safetensors"

# MODEL_ID = "TheBloke/WizardLM-7B-uncensored-GPTQ"
# MODEL_BASENAME = "WizardLM-7B-uncensored-GPTQ-4bit-128g.compat.no-act-order.safetensors"

# MODEL_ID = "TheBloke/wizardLM-7B-GPTQ"
# MODEL_BASENAME = "wizardLM-7B-GPTQ-4bit.compat.no-act-order.safetensors"

[sp1d3rino]

@dportabella hi, what is the response time for the best model you have tried till now on your GeForce 12GB?

[dportabella]

I don't manage to make it work. See this issue:
#510