AttributeError: 'NoneType' object has no attribute 'from_pretrained'

[Sosycs]

Thank you for such work!

I have been trying to use the Library for image captioning. by follwoing the instructions in @ouhenio comment on this thread: #313

I am using google colab pro and did the follwoing:

import os
from transformers import LlamaTokenizer, LlamaForCausalLM

save_folder = "/content/llm" #created a new folder to save the model and the tokenizer into

tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.save_pretrained(os.path.join(save_folder, "tokenizer"))

model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model.save_pretrained(os.path.join(save_folder, "model"))

next:
installed Fastchat using:

pip3 install --upgrade pip  # enable PEP 660 support
pip3 install -e ".[model_worker,webui]"
git clone https://github.com/lm-sys/FastChat.git
!pip install transformers accelerate

Then,
!python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

at last, I installed Lavis:

git clone https://github.com/salesforce/LAVIS.git
cd LAVIS
pip install -e .

after that I changed the line in "blip2_instruct_vicuna7b.yaml" into:

path to Vicuna checkpoint

llm_model: "lmsys/vicuna-7b-v1.5"

then I used the code:

import torch
from lavis.models import load_model_and_preprocess

device = torch.device("cuda") if torch.cuda.is_available() else "cpu"
model, vis_processors, _ = load_model_and_preprocess(
    name="blip2_vicuna_instruct", 
    model_type="vicuna7b", 
    is_eval=True, 
    device=device
)

and got the error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-6-3d26752714aa>](https://localhost:8080/#) in <cell line: 1>()
----> 1 model, vis_processors, _ = load_model_and_preprocess(
      2     name="blip2_vicuna_instruct",
      3     model_type="vicuna7b",
      4     is_eval=True,
      5     device=device

[/usr/local/lib/python3.10/dist-packages/lavis/models/__init__.py](https://localhost:8080/#) in load_model_and_preprocess(name, model_type, is_eval, device)
    193 
    194     # load model
--> 195     model = model_cls.from_pretrained(model_type=model_type)
    196 
    197     if is_eval:

AttributeError: 'NoneType' object has no attribute 'from_pretrained'

what am I missing? is it the folder I download the llama2 toknizer? or the path to the Vicuna checkpoint?

[Sosycs]

I have also downloaded llama 7B wights and when I try to covert to hf I got the error:

You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Saving a LlamaTokenizerFast to /content/llama_7b.
Fetching all parameters from the checkpoint at /content/llama_7b.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/convert_llama_weights_to_hf.py", line 407, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/convert_llama_weights_to_hf.py", line 396, in main
    write_model(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/convert_llama_weights_to_hf.py", line 178, in write_model
    f"model.layers.{layer_i}.self_attn.k_proj.weight": permute(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/convert_llama_weights_to_hf.py", line 154, in permute
    return w.view(n_heads, dim1 // n_heads // 2, 2, dim2).transpose(1, 2).reshape(dim1, dim2)
RuntimeError: shape '[32, 2, 2, 4096]' is invalid for input of size 16777216